They are both good camera's
The Fuji has a larger APS-C size sensor which is better for low light the Olympus has more features..
http://cameradecision.com/compare/Fujifilm-X-T10-vs-Olympus-OM-D-E-M10-II

I've copied this excellent run down of the main contenders by EIGofre from a post in another thread.


(Warning: Long post incoming)

For a long term investment around the £500 mark (with an initial spend of up to £600), there are four mirrorless cameras I would be looking at:

*Fujifilm X-T10 with 16-50mm lens for £468 (£548 minus £80 cashback)
*Olympus OM-D E-M10 II with 14-42mm & 40-150mm lenses for £494 (£569 minus £75 cashback)
*Panasonic GX80 with 12-32mm lens for £449 (£549 minus £100 cashback)
*Sony A6000 with 16-50mm lens for £499 (no current cashback offer)

All four have their own advantages and disadvantages. Firstly to cover IQ and portability you've asked about specifically- The Fuji and Sony have an image quality edge, but all four are excellent and, as a new photographer, you will be a larger bottleneck on image quality than your camera will be! For portability, click here for a size comparison of all four models. They're all quite similar, with the Sony and Panasonic being a bit thicker with their protruding grip/viewfinder, with the Olympus and Fuji are taller with their viewfinder humps. They're also a very similar weight (san-lens), with the Sony, Fuji and Olympus all weighing between 380g and 391g, with the Panasonic only a bit heavier at 427g.

So with those two factors covered, as a long term purchase there are other things to consider;
*Lenses: Panasonic and Olympus both use the same lens mount and can freely use lenses made by each other, and third part lenses will obviously work for both cameras too. This means a lot of variety and generally quite affordable prices. Sony and Fuji use their own lens mounts, and the options available are less diverse and often a bit more expensive. The Olympus also gets a big value bump in that it comes with a telephoto lens at the same price that these other cameras only have the kit zoom lens.
*Autofocus and continuous shooting: All four cameras have excellent autofocus for static shots, to the point where it's not really an issue I would make a choice on. However when it comes to subject tracking, the A6000 remains uncontested as the best mirrorless camera at this price point for focussing on moving subjects (it also does it with a continuous shooting rate of up to 11fps, the fastest of all four cameras). The Panasonic takes second place, while the Fuji and Olympus struggle in this regard (all three of these cameras shoot at between 6fps and 8fps in continuous mode, which is very respectful but nothing close to the Sony)
*Image stabilisation: Both the Panasonic and Olympus cameras have image stabilisation within the camera body, essentially turning any lens into an image stabilised one and making it easier to get sharp shots at slower shutter speeds. It also means lenses can be smaller and lighter when IS doesn't need to be built in. The Sony and Fuji do not have this feature in-body and it must instead be built into the lenses.
*Video quality: The Sony produces the best 1080p video of the bunch, while the Panasonic also has great video quality, can shoot all the way up to 4K, and the in-body image stabilisation means you can get smoother video regardless of the lens you use. Neither the Fuji or Olympus are really worth considering if video is a big priority, on account of poor subject tracking and a lack of video-oriented features.
*Viewfinder: All four cameras feature an electronic viewfinder of a good quality. All are perfectly usable, but in order of quality they go Fuji>Olympus>Panasonic/Sony.
*Styling and ergonomics: Obviously there's a lot of personal preference here, there's the more conservative design and rangefinder-style Sony and Panasonic versus the retro inspired SLR-style Fuji and Olympus. Two big considerations here though are the viewfinder placement and grip- the viewfinder of the Panasonic and Sony are placed near the left hand side of the camera, which is fine for most people but can make it awkward for people with a dominant left eye as you have to hold the camera across your face, while the centred viewfinders of the Olympus and Fuji are better in this regard. The grip also plays a big part for some people, with the small "finger holds" of the Fuji, Panasonic and Olympus being less suited to people with larger hands than the more robust grip of the A6000.
*JPEG image quality: This is not something normally mentioned in comparing high end cameras, as one of the big draws of these cameras is the editing and tweaking they allow you to do with images by shooting RAW mode. However Fuji have found a niche for themselves with their excellent JPEG presets and film simulations, and it deserves a mention here.

So yeah, ramble over. In short, all four are excellent and whatever you buy you will most likely enjoy and be very happy with the results of. However there are a few significant factors that will make some cameras better suited to some people but not others. Want an affordable camera and lens options from day one? The Olympus is a great shout. Want something you can take to sporting events or just for kids running around the garden? The A6000 will keep up with them. Want something stylish and will produce great photos out of the camera without worrying about editing? The Fuji is awesome for that. Want to dabble in videography? The GX80 has you covered the best.

EIGofre

There is a review here that also gives a good comparison of the two:-
http://thewirecutter.com/reviews/best-mirrorless-camera-under-1000/

Can't vouch for the Olympus but can recommend the Fuji - Photos taken are sharp and have good colours/contrast straight out of the camera. The user interface, EVF and controls are very good and the camera is a pleasure to use.

The image stabilisation is built into the lens with Fuji (same as Nikon). It works really well.

Can't see the price going much lower than this. This works out cheaper than a factory Refurb.

Got mine within a couple of days. But I did spend a little more and go for the nicer looking silver version.

Received mine OK. Took over a week due to courier ****ups. Very pleased have taken some excellent shots already.

Delivery was via DPD/Interlink, no problems with Camera World.

It might pay to check with Fuji. This model is no longer showing on the Fuji store so may be discontinued now.