Yeah but big price difference too so gotta weigh up the pros and cons

There are four versions now as you may be aware. The mk2 was a relatively mild change and you can get away with choosing the original version to save cash. The mk3 version brought a more radical change, including a lens with a far larger aperture when zoomed in (this is known as being a "faster" lens, which let's in more light and let's you use faster shutter speeds and creating shallower depth of field) and a pop-up electronic viewfinder for composing your photos with. The mk4 was again a more incremental change, primarily focusing on adding extremely high speed video and shooting modes.

As a casual/hobbyist, the original is an excellent camera that you'll be extremely happy with, the further versions add tempting features for snthusiast users but for using as a conventional compact camera, you can't go wrong with the mk1.

It's all about sensor size I believe. The sensor in this camera is really large for a compact camera. Mega zooms have smaller sensor sizes to allow them to have such short lenses, you just have to look at the length of lens required on a pro level DSLR to see what you'd need for a 10x on a decent sized sensor camera..

On the other hand, I have a (now 10 year old!) mega zoom which takes fairly good photos, and has a 12x or so zoom. It is a 5 megapixel camera, allowing 6x4 (possibly larger) to be printed out.
This is a 20mp camera, with 3.6x zoom. With some cropping, you'd have a photo which could easily be printed to 6x4, probably larger, and the equivalent of a lager zoom. I'm looking to replace the old mega zoom with an RX100 at some point with this line of thinking...

It's less a matter of innovation at this point and more a matter of physics.

As mentioned above, sensor size is a major factor in how good a camera's image quality can be- the larger the surface area, the more light the sensor takes and the more information it has to work with in a given exposure. This is the big limiting factor on why bog standard compacts don't have particularly good image quality- their sensors are tiny. However the physical properties of the sensor in turn dictate some of the physical properties of the lens, for example it needs smaller pieces of glass in the lens to cover the sensor fully, the lens can be smaller while retaining a usable aperture etc. It also means a small lens can create a large degree of magnification, also known as crop factor. This makes tiny sensors ideal for consumers whose primary concerns are convenience oriented rather than chasing the best image quality they can get, because it means designers can create lenses with enormous zoom ranges in a comparatively tiny space, while simultaneously being retractable into a pocketable body.

When you move to a larger sensor, the lens needs to get larger too- you need larger lens elements to prevent vignetting and to retain decent apertures, and all of this requires more physical space. The RX100 series are a good illustration of this- the mk1 and mk2 versions of the camera had a 3.6x zoom lens but when fully zoomed out the maximum aperture was a fairly low f/4.9. In the mk3 and mk4 versions they used a new lens with a much larger maximum aperture of f/2.8 when fully zoomed out, which lets in a lot more light, but they had to reduce the zoom range to 2.9x in order to stay in the same physical dimensions.