Fenix A320: Anniversary

Posted May 19, 2023

Hi all,

Today marks 1 year of Fenix. It has been a remarkable journey and an incredible honour for the team to be allowed the privilege of not only sustaining, but growing the business and team behind the A320. We owe that thanks to our customers, you all, for supporting us throughout our endeavours - so from all of us on the team, a hearty thank you!

Now, as much as looking backward is fun, all of you are also, no doubt, keen to know what lies ahead for the A320. Today felt appropriate to begin slowly lifting the covers on our next major update to the aircraft, and to begin showing you all what to expect.

As some may know, we made the difficult strategic decision to pause work on short-term updates in lieu of a larger, broader, and more substantial single update addressing a number of fundamental issues and improvements to the product. This would not have been possible to do alongside the process of constantly patching the public version of the A320 in parallel, given the substantial infrastructure and core-system refurbishments required. While I understand this may be frustrating in the short term, we are also incredibly happy with the results of this painful process - and truly believe that there was no quicker, or better way of producing these necessary upgrades for you.

So, let’s get to the meat of it.

The IAEs. I have personally been haunted by the chorus of “IAE wen” for the last 6 months, so I am very happy to say that the IAEs have now been pushed to beta, featuring our proprietary external engine model technology. To build our own physics model for turbofan engines and network it into the sim and aircraft was a significant undertaking, numbering in the thousands of man-hours worth of work - data collection, data organisation, data interpretation all came first. This in itself was a feat, as the degree of accuracy we were aiming for required that we have concrete, real life, data points to build the model around. Thousands of them, ranging from oil quantities, fuel flow, N1/EPR vs. OAT, to ignition probabilities during windmill starts, right down to turbine core heat vs. EGT data, and even heat dissipation data. Making some data up or “filling in the blanks” with a guess would have caused the entire model to collapse like a house of cards, as they all relate to one another. Organising thousands of data points was also a heavy-duty task, as these needed to be laid out in a way that made sense for reference and use. And only thereafter, we started digging into and understanding how the engine behaves and reacts to different atmospheric conditions, and thereafter, differences in its own parameters.

Once all that was complete, we designed, built and prototyped the infrastructure for the model. Building a framework for how the data and model interacts with itself, then its environment, and then subsequently the flight dynamics engine. This was where the majority of time was spent, making sure that the various transient states of the engine were modelled smoothly, and accurately. We redesigned the infrastructure several times over, adjusting for performance (both, computational and behavioural), but without sacrificing the granularity that the model afforded us. Significant efforts were made to achieve a performant solution without sacrificing data resolution.

This development was an eye-watering investment, both in time and money, but the end result is an infrastructure we can use across various different aircraft engines, type agnostic. It’s a cool piece of tech that should help future Fenix products as much as it will help the IAEs right now - and for that reason it was worth pursuing.

But, talk is talk and.. Well, it isn’t as interesting as a small demonstration. The objective here is to show you all the physics behind the engine model as best as possible, so they will be relatively specific scenarios, but I hope they get across the level of dynamic detail we have pursued with this development. Please note, this is work in progress footage and software. Things may not be perfect, we are still very actively working on the physics model.

Until the engine model is 'signed-off' as pretty much good to go, we can't begin proper implementation of the sounds - for anyone wondering why there's a lack of IAE noises in the videos.

What you see here is an interaction between the thermal model of the engine’s core, the modelling of the EGT probe, and airflow modelling of the N1 and N2 fans, demonstrating thermal transfer properties between the core, the air mass through the engine (and around it), and the probe. The EGT probe sits on a strut toward the exhaust shroud of the engine, whereas the core is naturally nestled within the actual turbine itself. Your EGT reading in the flight deck does not, therefore, tell you about the temperature of the core - it simply tells you about the temperature of the exhaust gas. However, the exhaust gas is influenced by the turbine core temperature, the airflow through the core, and out the exhaust and therefore effects like the one you see above occur.

So what is actually happening? Well, what you’re seeing is a shutdown. Seems simple, right? Engine off, so EGT goes down. Not quite, as you can see. Observed on IAEs in the real world, after a fairly rapid drop in EGT reading, it actually stops and starts climbing again, before stabilising, and beginning its progressive drop toward OAT. So why does this happen? As the N1 and N2 fans stop rotating they stop driving air through and around the core; This leads to a relatively quick drop in the EGT probe’s readings as the exhaust air quickly becomes replaced with ambient air around it - however, the core (being slower to cool) continues to progressively diffuse its heat toward the EGT probe, eventually reaching a heat transfer equilibrium, and then thereafter actually heating the probe up again! Eventually, as the core also begins to cool, the entire system stabilises and finally begins losing heat, trending towards OAT, but this time slower - as the combination of core heat, the EGT probe’s thermal model, and OAT begin playing with one another having all settled toward the cooling bias. Pretty nifty stuff (and hideous to code dynamically rather than as a scripted event..)!

Until the engine model is 'signed-off' as pretty much good to go, we can't begin proper implementation of the sounds - for anyone wondering why there's a lack of IAE noises in the videos.

Now for something a little bit different. In this demonstration, we’re starting an engine after a very short turnaround. As we start the engine, you can see the residual EGT is sitting at around 250 degrees. As the N1 and N2 fans begin rotating for the start process, they begin driving air through and around the core, slowly driving compressed and core-heated air back towards the EGT probe. This leads to a gentle spike in EGT reading, before the fans begin picking up speed, and the volume of fresh, cold air being ingested by the engine overcomes the immediate heating and cools both the core and the EGT probe, depending on the temperature (sometimes it cools, sometimes it doesn’t!). It’s a much faster process than the shutdown as we’re now driving cold air in a deliberate manner through the engine, via the N1 and N2 fans. The effect of this is of course varied with OAT, and is entirely dynamic depending on the temperature conditions of the ingested air and temperature of the core, so sometimes it will simply heat up slightly before ignition takes place and there will be no cooling effect.

Additionally, in high temperature conditions, or short turnarounds - the EGT and core will remain fairly hot even after a full turnaround. The FADEC will therefore command the IAE engines to motor themselves to cool the engine before initiating the start process. You can see this process taking place, as ignition is delayed until EGT is reduced to at or below 250 degrees. The rule is that ignition is commanded when the EGT reads at or below 250 degrees, or 150s after starter engagement, whichever comes first. Here you can see the cooling effect of the motoring, driving temperatures down using similar thermal principles.

A fun detail we didn’t get a video of, but that our devs wanted you to know about (I was directly told this needed to be included), was that core heat actually interacts with the oil temperature, even. Oil flows inside the engine veins, and is influenced by the heat of the core. When the core is hot, the oil will carry some of that heat with it as it flows through the veins and back out again. Ultimately, the teeny tiny detail of going through the hassle of modelling this behaviour and subsequent thermodynamic interaction is when you shut the engines down, the oil temperature will simply be higher than when you first started, and will then cool over time. But it won’t just fall to the temperature it was before you started. Cool little details make the whole thing come alive.

That just about wraps up the engine demonstrations for this post - but I hope it goes a good way toward showing you the level of detail we’re working towards. So far, we’ve been satisfied with the accuracy of the fuel flow, as reported by the beta team, however there remains some tweaking and tuning with thrust outputs that need to be done here - followed by some failure implementations, which we will talk about closer to the release.

//

For those eagle eyed amongst you, our new display graphics and fonts were front and centre for those demonstrations. We have completely thrown away the old display graphics and fonts and began the process of rebuilding them from scratch for absolute accuracy, as there were a fair number of complaints about various elements being out of proportion, fonts being incorrect, and shapes being inexact. After a careful review of these complaints and bug reports, we determined that the issues with the old displays warranted a complete rebuild as they were fundamentally flawed and could not be rescued, and the cleanest way to satisfy the stringent requirements of our customers was to start again, in-house, instead of using the inherited graphics and font set. We set about acquiring hundreds of high-resolution, tailored, reference images of the actual displays on board real aircraft via our fantastic network of technicians and pilots, and set about building bespoke fonts, and drawing all graphics, pixel by pixel. We’re very pleased with the result so far - but work continues on this project, with further nips and tucks required to font thickness and bolding amongst other minor details, along with the full completion of the various other display elements.

For now, we would like to present a quick and dirty WIP before-and-after to demonstrate the substantial difference between the old and new set of displays.


Old PFD.
New PFD.


Click and drag your left mouse button on the ECAM to see the difference. Left side is old, right side is new and improved.

To be completely honest, before undertaking this, I was of the mindset that the differences were numerous but miniscule and did not warrant such an enormous amount of work, and that the complaints were a mountain out of a molehill - however, upon reviewing the data and the before/after images, I can quite confidently say that I was wrong, so for all those who submitted reports about this - bravo, and great eyes! The details certainly add up, and the displays now look fantastic and quite substantially more accurate. This also leads me quite neatly into the art and performance section of this update, as in addition to rebuilding the front end of the displays, we’ve also gone and rewritten the render system behind it, to further optimise the performance and stability of the product.

//

From an art perspective, a major point of this update will be to achieve two things:

  1. A noticeable and meaningful performance boost

  2. Reduce the VRAM/memory consumption of the artwork, whilst somehow increasing the detail/accuracy

Point 1. A little background on why Fenix performance is worse than some aircraft...

The issue stems from a choice we made back in 2020, when the MSFS SDK did not support the type of advanced display drawing technology required to accurately or performantly represent a modern airliner. From talking to other developers at the time, this was a major roadblock for them too. As we weren't sure how long the wait might be for better support, and all communications at the time pointed towards "eventually", we took the leap to design our own system over the course of several months. This proved very successful, achieving better performance than even the default A320. Given this success, we saw no reason to change away from our display system, even as MSFS started to support more complex internal display rendering. As we began to lock-in development and push towards beta, MSFS received a much needed substantial performance update that pretty heavily reworked the resource utilisation of the software. However, it near-enough inverted our performance lead. After a few months of significant optimisation work we were back within margin. Post-release it became clear we were not meeting user's expectations when it came to performance, so we entered another round of heavy optimisation over the summer of 2022. At the end of this, it became clear we'd squeezed everything we could out of the current rendering tech, and if we wanted more it was going to be a start-from-the-ground-up job. I'll be honest, it's never a fun day when a developer gives you an estimate for a single-ticket item in "months/years", especially when that item is possibly the most important on the list. This is why it hasn't been fully addressed over the last dozen or so updates, even if we've managed to chip away a few edge-cases in the meantime.

How are we addressing it?

We started by investigating all applicable display rendering methods, both internal and external; evaluating them for performance and capability (such as the complex drawing and masking required for the A320 displays). From there, we explored how long a 'switch' to any new method might take, whether we need to hire extra resources or specialist developers, and whether there could be any technical issues with any given rendering method. This whole process took a few months, but it left us with some extraordinarily useful data that gave us a clear way forward. Since late last year we've been building upon this method and are really happy with the results. We think there's still some gains to be made so we're not ready to share benchmarks or figures just yet, but we have seen some very encouraging results from testing.

Point 2: VRAM/memory usage & visuals...

I'm going to try and keep this a bit shorter, but on top of our rendering system we also noticed some customers experiencing further performance degradation relating to VRAM overflow. This is where there's just too much stuff trying to be loaded into video memory, and the graphics card ungracefully hands it off to system memory, and becomes a little princess asking the CPU to do all the extra fetching for it. The bigger the overflow, the more stuff the CPU needs to handle that it really isn't supposed to, the less time it has to take care of its already busy workload. Man, it's starting to sound like Fridays at Fenix HQ. Anyway, there's really only two ways to solve this performance hit: More VRAM or giving the GPU less memory to store. I don't think it'd be a good look for us to tell our customers to "just go buy a 4090", so instead Dave and the art team have risen to the challenge.

How are we going to reduce VRAM usage without making it look worse?

To start with, the art team did a hefty amount of benchmarking to understand where that memory is coming from (note: memory on disk rarely reflects memory in GPU), we looked at everything from vert counts to animations, from clickspots to material switching. After a lot of testing we ended up with a clear 'hit list' to reduce the memory usage in a way that wouldn't negatively impact the visuals. This involved rebuilding the exterior model almost from scratch, and tearing the cockpit down to its basic underpinnings, then rebuilding both with our new understanding of MSFS resource utilisation. Along the way we were able to improve the accuracy and detail of the exterior model significantly, and through some crafty new texture mapping we've reduced the cockpit memory usage by over 50% with a slight boost in quality/detail. The overall savings are yet to be seen as we still have quite a bit of texturing left, but it's already having a measurable positive impact on lower-end systems.

Anyway, that was a heck of a lot of words and the most exciting bits of V2, for me at least, are elsewhere. For example, shadow-casting lights that have 0 performance hit compared to standard/default lights:

V2 Lights

For the purposes of this demonstration, we’re using the V1 exterior model and textures, but here you can see both the wing light, and the landing light and casting shadows. Bear in mind, the light intensities are incorrect, as the majority of the work so far has been on developing the tech to power shadows from MSFS dynamic lights. Given the wing light is slightly further aft of the camera position, it’s also casting a shadow to the inside of the engine inlet and across the N1 fan. Taxiing out to the runway, the wing and engine subtly moving about as you cross ruts in the tarmac, watching the lights come to life one by one casting unique shadows and lensing effects, really is something to see.

V2 Lights

As always, these images are heavily work in progress so lots to add/fix/change, but here you can see the lower beacon casting shadows away from the main landing gear, and if you look further out even the engines. You can also see the new logo light with the unique lensing that occurs on the A320, causing this sort of ‘double light’ effect at the bottom and top of the tail - go check out some photos of the real thing - we’re getting very close here!

We’re saving the most exciting bits for another time, but that’s your first taste of what we’ve been cooking in the art department for V2.

Additionally to all the above, the flight model itself has not been forgotten, and a considerable amount of work has been done polishing off certain phases of flight that were suboptimal. This itself took a few hundred hours of investigation and tuning to ensure the behaviour was just as desired, but we hope you all feel these improvements in your day-to-day flying experience.

The fixes were targeted at improving a few specific and critical flight regimes, namely:

  • Crosswind Takeoffs

  • Crosswind Landings

  • Takeoff pitch input for rotation

  • Landing Bouncy Castle Super Fun Times

I will expand on each in detail, along with the noted/associated issues. Please note, all videos seen here are using old displays, old textures, and a dev build with potentially weird stuff going on in the background.

Crosswind Takeoffs

These were incredibly unstable, and often required a "two step" rudder approach, once when rolling down the runway, followed by a seemingly violent increase in required rudder to maintain the centerline once rotation had begun and the nose wheel lifted into the air. The sudden jab of rudder would induce significant roll, moreover, the action of increasing rudder input during rotation was incredibly unintuitive and caused a lot of undesirable handling qualities immediately post rotation.

The issue stems mainly from requiring rudder input during rotation in the first place, whereas in reality, the aim would be to gently release the rudder during rotation. Unfortunately, requiring rudder on rotation is more or less hardcoded into MSFS due to the way wheel friction influences the aircraft track - multiple tests were conducted at varying lateral friction values, concluding that the MSFS physics engine needs a fair bit more work in this transitional phase of flight. Despite the aircraft correctly weathervaning into the wind and the main gear pivoting accordingly, the expected behaviour is that the aircraft would continue along it's flight path - however, in MSFS, the pivot of the main gear would force the aircraft to track in the direction of it's wheels as opposed to the flight path. No combination of values or parameters can influence this. Instead, work was focused on improving the predictability and intuitiveness of the rudder application, and combined with improving the fly-by-wire system to correctly and quickly react to the roll moment induced by the application of rudder.

At this time, the solution has resulted in what we believe to be a much more stable post-rotation environment. A couple of videos are attached showing a before/after in similar conditions.

BEFORE:

To note, once the nose wheel has lifted from the ground, the aircraft immediately and violently starts deviating from the centerline toward the edge of the runway. A massive rudder input increase (visible in the debug menu as "SimVar Rudder'') is then required to prevent the main landing gear from exiting the runway, and despite this large input increase, the aircraft is still unable to completely overcome this force and maintain the centerline. Once the aircraft's main gear has lifted from the ground, the large rudder input causes significant and violent roll, which is not reacted to at all by the fly-by-wire system, allowing the aircraft to roll onto it's side, effectively. The subsequent sidestick input to correct this roll is also relatively unintuitive, requiring different degrees of sidestick application depending on the blending phase of the aileron.

CORRECTIVE MEASURES:

The following actions were taken to combat this behaviour -

a) Rudder authority retuned system side to provide consistent behavioural output to the user.

b) Fixed a bug with aileron ground-to-normal blending law, now correctly blending in 0.5s per the real aircraft with retuned response times during blending.

c) Retuned and refined fly-by-wire implementation re Yaw x Roll relationship, the aircraft should now more appropriately compensate for roll moments induced by introducing rudder input.

d) Retuned lateral ground friction values in conjunction with the above rudder authority changes to ensure cohesive and consistent lateral control properties.

AFTER:

Once the nosewheel has lifted from the ground, the aircraft's track deviation is substantially reduced, and the rudder application required to maintain centerline is unchanged. As the aircraft leaves the ground, no adverse roll is noticed and no violent sidestick aileron action is required to maintain the wings level. Once the main gear has left the ground, rudder input can be gently released and the aircraft weathervanes into the wind, once again requiring no aileron input from the sidestick.

Crosswind Landings & Landing Bouncy Castle Super Fun Times

The correct procedure in an A320-family aircraft is outlined in various places, however to recap quickly, the correct method involves decrabbing the aircraft during flare and aligning the nose of the aircraft with the centerline before the main gear touch down. Doing so requires squeezing in rudder to decrab, with a dash of aileron to maintain tracking in the correct direction. The fly-by-wire system would compensate for any further aileron required due to the rudder input required during the decrab procedure, theoretically keeping the wings level during this procedure.

Decrabbing the Fenix was difficult in this context, as the rudder input to decrab the aircraft would cause wing-dips and the aileron control authority would seemingly be non-linear and fairly unintuitive due to the varying degrees of response, seemingly random in nature. Once the aircraft contacts the ground, it would often bounce, followed by an excursion to either side of the runway due to the lateral wheel friction issues noted above. All in, a bit of a nightmare, and a very unpleasant experience.

We believe we have now solved this issue in relative whole, with the concession that we have improved but not fixed the lateral friction values on the ground due to the hardcoded behaviour of the tires. Overall, however, a significant improvement in the aircraft's behaviour is seen.

BEFORE:

Note the application of rudder during the decrab procedure causes the aircraft to roll to the right, despite gentle left aileron being applied. After a strong aileron correction, the aircraft subsequently lands on one wheel first, bouncing, before settling, and tracking far from the centerline as a result. All in, a rather messy experience, with violent and relatively unintuitive control inputs required to land the aircraft safely.

CORRECTIVE MEASURES:

The following actions were taken to combat this behaviour -

a) Fixed a bug with the fly by wire inhibiting automatic aileron orders being sent when the aircraft was in FLARE mode.

b) Rudder authority retuned system side to provide consistent behavioural output to the user.

c) Reworked spoiler deployment to dump lift faster, ensuring the aircraft settles quickly.

d) Retuned and refined fly-by-wire implementation re Yaw x Roll relationship, the aircraft should now more appropriately compensate for roll moments induced by introducing rudder input.

e) Retuned lateral ground friction values in conjunction with the above rudder authority changes to ensure cohesive and consistent lateral control properties.

AFTER:

The rudder input to decrab the aircraft is compensated for by the fly by wire system, it correctly and appropriately applies aileron input to keep the aircraft wings level during the procedure. The aircraft touches down and settles quickly, and a consistent and intuitive amount of rudder is required to reacquire and track the centerline, beginning from flare to rollout.

Takeoff Pitch Rotation

The airplane rotated far too easily and with too little backstick involved. The problem was traced to an incorrect elevator limit calculation, this has now been corrected and the pitch input required to rotate the aircraft has changed, and in my opinion better reflects the IRL aircraft.

Summary

I spent too long doing circuits and got very dizzy. But it should be better now.

//

We have also built in a number of avionics changes and fixes, the changelog of which is slightly too long to post - however we’re working on fixing a number of common bugbears with VNAV, DECEL point calculations, general system behaviour errors, hydraulic system improvements, issues with autoflight missing descent constraints, FMA improvements, crash fixes, so on and so forth. For this, there is little to describe - but rest assured the avionics side of things has not gone untouched during this period also.

//

Next on the list of massive overhauls, the EFB.

Before starting, we'd like to say a big thanks and acknowledge the tremendous work put in by the original EFB's creator, Katie, who is moving onto the next step in her flying career, and all of us at Fenix wish her the very best with it! Katie will be continuing to help the Fenix team with feedback and support, as she passes the torch to Luke, a new member of the Fenix team who will be focusing his efforts on bringing further improvements and additions to the EFB. Luke, alongside his dedicated team, are thrilled to unveil some significant performance and quality of life improvements to the EFB.

To start with, we have rewritten the foundation of the EFB. The team has migrated from a pure Javascript and PHP codebase to a more modern Angular front-end application powered by a C# backend. This transition not only allows our developers to work seamlessly on an industry-standard codebase, but primarily accelerates the development of new features for the EFB with greater flexibility and extensibility.

One of the most noteworthy enhancements as a result of this rewrite is the lightning-fast navigation between "apps”. This has subsequently allowed us to remove the transition effects between page loads, as they proved to be slightly cumbersome within the simulator environment. The speed and responsiveness of the new backend has created a more fluid experience overall, cutting down on loading times.

Please note, the keyboard is in need of some scale adjustments to make it easier to read/user-friendly.

The performance improvements continue within each app too, as we've made major enhancements in the calculation of v-speeds and landing data. We've shifted the calculation logic from the cloud to the client side (your computer), resulting in nearly instantaneous calculations. It’s so fast, we’re considering adding an option for users who would prefer some sort of a delay for the sake of immersion.

There is also now the ability to add a custom runway intersection (a much requested feature!). If the intersection you’d like to depart from isn’t already in the EFB you can simply add your own, and your v-speeds will now be calculated using the runway length available to you.

Based on user feedback, we've made a few additional quality-of-life improvements, including:

Option to disable the lock screen. Reduced the time it takes to unlock the lock screen with the fingerprint. New virtual keyboard, with physical keyboard input support. You can now directly type into the boxes if you prefer. Virtual keyboard auto-closes itself after a valid value has been entered into the input field. Improved responsive styles - supporting more display sizes and resolutions. Some users reported the UI not working correctly, now it will! Did somebody mention IAEs? IAE performance data has been added for the calculators.

There’s a lot more to come for the EFB. As always, we value your feedback and encourage you to share your thoughts on what you’d like to see once this next version of the EFB has dropped!

//

Finally, I’d like to thank you all for your patience. As you can see, an absolutely gargantuan amount of work has taken place in the long wait that you have had, and I am very hopeful that the Fenix A320 you receive when this update does drop will be a significantly overhauled and improved version of the aircraft you have today. I hope this little (big) update post has been enough to stay the hounds for a while yet.

You’ll hear more from us closer to the update landing. For now, have a good day, all!