Tuesday, November 09, 2021

Light and Weather in Microsoft Flight Simulator

Fans of flight simulators often talk about how “immersive” a sim is. It’s about how much the sim experience makes you feel like you’re flying a real airplane. Visuals are obviously a key component, and VR can take this to a higher level in terms of feeling truly surrounded by a 3D environment made up of the structures of your aircraft as well as all the natural and human-built objects in the simulated world around you. Sound is also very important, as are physical controls like yokes and rudder pedals (controlling an airplane with a keyboard is definitely an immersion killer). Realistic ATC (Air Traffic Control) communications are important especially for airliners and military flight sims.

Microsoft Flight Simulator 2020 addresses all these components, but to me, lighting and weather are the pieces that really pull it all together. Things we take for granted in real life like shadows, reflections, transparency, refraction, and color shifts are all well-simulated in MSFS. Making things look real also requires careful attention to materials and to how light interacts with them, including their colors and how shiny or rough they appear to be, which also affects color.

MSFS uses techniques called PBR (physically based rendering) to pull this off. Notice how the yellow features in this Stearman paint scheme (above) are reflected and distorted in the shiny blue fabric panels of the wings. The colors are shifted by these reflections as well as by shadows and by the angle from the sun at which you are viewing the surface. PBR techniques are also used to make buildings, runways, and other surface objects appear more realistic, often in combination with photogrammetry. 

Photogrammetry uses photographic imagery for measurement purposes, and in MSFS, this real-world data allows many cities to appear nearly photorealistic. Based on the resolution of the captured data, objects as small as signs on buildings may be readable, depending on your position and altitude. Technologies such as radar and lidar can also be used to accurately measure the height of terrain, and such data is often available to simulation modelers. Although MSFS recreates the entire surface of the Earth, the level of available detail varies by location, with many major cities covered by detailed photogrammetry (like New York City, shown in a VR stereo pair below), while many others depend on AI-based “autogeneration” of 3D objects including buildings, mountains, trees, etc. 

The appearance of objects in a simulation naturally depends on light sources, and MSFS incorporates multiple light sources starting with the sun and the moon (both accurately positioned based on the location, date, and time of day). I believe starlight is also modeled. Human-made light sources include streetlights, vehicle lights, light from building interiors, airport lighting, etc., and what amazes me is  when light sources interact with surface properties (reflecting or scattering) as well as with atmospheric properties like clouds, mist, rain, and snow. The resulting scenes can be uncannily realistic.

Speaking of rain and snow, weather is the other big piece of “immersiveness” I wanted to briefly discuss here. It’s a critical subject in real life aviation and in the sim. Microsoft and Asobo and their partners have placed huge emphasis on making weather look and feel right, and the system they developed is incredibly detailed, flexible, and easy to control.

Not only can you adjust any location's time and weather to your choosing - from clear skies to heavy clouds, to snow and storms – with “Live Weather,” you can even set it to match real conditions taking place in that part of the world. But if you don’t like the time of day, the wind, the height of the clouds, the height of the waves, or the depth of the snow cover, you can change any of that without even stopping your flight, using the interface shown above. 

Rather than drone on and on about this, allow me to recommend two brief developer videos that clearly explain how they managed to make the weather so real, it can get scary at times. Fortunately I can switch to clear skies anytime and (usually) not hit the mountain that was behind that cloud!

Feature Discovery Series, Episode 2: Weather 

Partnership Series: Meteoblue - Weather Forecast System

Monday, November 01, 2021

Why Things That Fly Are Cool

I blame my main life obsessions on John Glenn and the Beatles: things that fly and making music. The flying bit started first, in 1962. In February of that year, John Glenn became the first American astronaut to orbit the Earth. I was nine years old and for reasons I don’t fully understand, the whole space thing was immediately very exciting to me. I must have watched the launches on TV and seen the pictures in LIFE Magazine. I started reading everything I could about the astronauts and soon learned that they were first military jet pilots. So I decided that’s what I would be. I’m sure around a million other American kids decided the same thing. The Beatles and making music came a few years later.

I stuck with my Air Force Academy, jet pilot, test pilot, astronaut dream for around three years until it was shattered by myopia. I get that military pilots must have uncorrected 20/20 vision, and even though I was too nearsighted to fly for the Air Force, my interest in space and aviation continued. I decided I would be an aeronautical engineer. I read books and built dozens of model airplanes, and when I was around 12, I joined Civil Air Patrol (CAP). It was something like boy scouts sponsored by the Air Force, but even better – they had airplanes! Cadets could go on demonstration flights in a Piper Cub! They would even let you fly the plane! It was the best thing ever.

CAP faded in high school as I developed other interests (e.g., the Beatles) and my parents got tired of schlepping me to the airport. I still built model airplanes and loved everything about flying, and I still assumed I would study aeronautical engineering and get a private pilot’s license as soon as I could. Of course, “life happened” and I tried a few different majors, quit college for a while to try being a singer-songwriter, and ended up studying physics and optics instead of aerospace engineering. A lot of other stuff happened too of course, like a family and a career.  I eventually did get a private pilot’s license, but not until I was 48 years old. That was great, but for various reasons, I couldn’t fly as much as I had hoped I would.

Which finally brings me to PC flight simulators. I played with a few in the mid-90s, military ones that were fun but not very much like real flying. By 1999 when I started taking real flight lessons, PC flight simulators had improved enough to be useful for some aspects of flight training, e.g., practicing some specific tasks like VOR navigation. But they didn’t look or feel very much like a real airplane, and I only used sims for a few things like preparing for a cross-country flight.

For a couple of years, the flying bug seemed to fade, other than reading my monthly AOPA Pilot Magazine and listening in to the cockpit audio on my United Airlines flights. In 2005, I discovered the free space flight simulator Orbiter, which reawakened my interest in space and in flying, since some of the simulated spacecraft in Orbiter had wings, like the space shuttle. This “space phase” lasted for a few years. It launched this blog and led me to become a NASA/JPL Solar System Ambassador, a volunteer educational outreach program for space and astronomy. I’m still a sucker for a rocket launch or landing, and I would go on a space flight in a heartbeat if I could manage it.

Flight Sim Renaissance

Throughout the 2000s, flight simulators continued to improve, but I didn’t pay any attention until sometime in 2020 when I heard that an amazing new version of Microsoft Flight Simulator (MSFS) would be released in August. I started reading articles and watching preview videos, thinking that it might be interesting to play with, since like many people, I was stuck at home due to the pandemic. Early in 2020, I had also started to get heavily involved in orchestral composition with sample libraries, so I decided I would wait a while before trying MSFS 2020. I was working with a composition coach and had started entering some competitions.

But by mid-September, the MSFS videos I was seeing were just too exciting. It was clear that MSFS 2020 was something entirely new and different, using modern cloud computing to model the entire world in incredible detail and with beautiful graphic quality. Not to mention airplanes! So I bought it, and within a month, I had upgraded to a new computer and monitor to run it better. My music took a back seat.

Why Is This Still Interesting?

Human dreams of flight started long before the Wright Brothers, and my personal dreams of flight started early in my life. Since I was also interested in travel and in foreign languages, I guess I hit the jackpot with the job I recently retired from, which would send me on international trips 6-8 times a year over many years. I spent a lot of time over the Atlantic and Pacific. And I spent some 120 hours at the controls of small airplanes (including flight instruction time) – a “low time” pilot for sure, but I’m grateful for the experiences and the memories.

With all that, you might think a flight simulator would pale in comparison. It’s definitely not the same as real flight but it still holds my interest for a few reasons:

  • It exercises the same “muscle memory” as real flight – the simulation is accurate enough to follow similar flight procedures and see similar results, both good and bad.
  • The simulated world and systems are accurate enough to allow for realistic flight planning, including “Live Weather” based on real-world data.
  • If I choose to navigate visually (VFR) in a familiar area, my real-world knowledge applies directly, which is quite satisfying. So I can navigate by "IFR" (“I follow roads”).
  • I never learned to use advanced GPS flight navigation systems that have become widespread in recent years, but now I have – and the simulated systems are very close to the real ones, which is fun to experience. So I can navigate by IFR (“instrument flight rules”).
  • Using multiplayer features and communication software, I can fly and talk with other sim pilots from anywhere in the (real) world. On many group flights, I have toured national parks and various international destinations, adding some of them to my “must visit” list for future real travel. New Zealand is at the top of my list once it opens for international visitors. 
  • I can learn what it’s like to fly airplanes I could never experience in real life, from a Stearman biplane to an F-14 Tomcat to a Boeing 787. Is it exactly the same? Of course not, but the better models highlight the critical flight characteristics and flight systems, rewarding you for handling them correctly, and "punishing" you for mistakes (RTFM to minimize this).
  • The childhood model airplane builder in me appreciates that I can easily acquire a collection of visually stunning airplane models that I can also fly in the simulator, unlike the plastic models from years past (though I still have a few die cast models of favorite airplanes).
  • As I mentioned in an earlier post, virtual travel is also a big attraction for me. It’s especially fun to fly around cities I know well from years of business travel, especially Tokyo, Paris, Seoul, and London. I like to recognize familiar sights and discover new ones I plan to visit.

Finally there is VR, a totally different way to experience simulated flight, where you feel like you are inside a cockpit and where clouds and buildings look 3D and startlingly real and massive mountains truly look massive.