Login or Sign Up
Login or Sign Up
Just made public, the patent was first placed in March 2020
Patents are always a rather intriguing thing to track in the technology industry – sometimes they represent ideas and products that’ll never see the light of day, but there are occasions when they give clues to future releases. A freshly released US patent application could be the latter, as it addresses technology that could allow Nintendo to boost visuals in its hardware.
Opened on 25th March 2020 and released publicly yesterday (30th September), the application is titled ‘systems and methods for machine learned image conversion’, and though the initial language can be a headscratcher it is essentially an idea similar to NVIDIA’s DLSS. That’s short for Deep Learning Super Sampling in NVIDIA’s case, which works on some of its GPUs to upscale image resolution and quality in real time, while also being remarkably efficient and ensuring that the graphics card is under less strain. It’s impressive technology, and it’s been front and centre in a lot of talk about how Nintendo could produce a new Switch-style portable device that outputs higher resolution visuals while still working with low power output. Digital Foundry has explored this in detail.
What makes this application intriguing is that Nintendo is clearly exploring this internally – a named party on the application is Alexandre Delattre, who is a co-founder of Nintendo European Research and Development. It’s also acknowledged in the ‘Introduction’ of the patent that this is an area being explored throughout the industry:
Machine learning can give computers the ability “learn” a specific task without expressly programming the computer for that task. One type of machine learning system is called convolutional neural networks (CNNs)—a class of deep learning neural networks. Such networks (and other forms of machine learning) can be used to, for example, help with automatically recognizing whether a cat is in a photograph. The learning takes places by using thousands or millions of photos to “train” the model to recognize when a cat is in a photograph. While this can be a powerful tool, the resulting processing of using a trained model (and training the model) can still be computationally expensive when deployed in a real-time environment.
Image up-conversion is a technique that allows for conversion of images produced in a first resolution (e.g., 540p resolution or 960×540 with 0.5 megapixels) to a higher resolution (e.g., 1080p resolution, 1920×1080, with 2.1 megapixels). This process can be used to show images of the first resolution on a higher resolution display. Thus, for example, a 540p image can be displayed on a 1080p television and (depending on the nature of the up-conversion process) may be shown with increased graphical fidelity as compared to if the 540p image were displayed directly with traditional (e.g., linear) upscaling on a 540 television. Different techniques for image up-conversion can present a tradeoff between speed (e.g., how long the process takes for converting a given image) and the quality of the up-converted image. For example, if a process for up-converting is performed in real-time (e.g., such as during a video game), then the image quality of the resulting up-converted image may suffer.
Accordingly, it will be appreciated that new and improved techniques, systems, and processes are continually sought after in these areas of technology.
Ultimately it shouldn’t be a surprise that Nintendo is researching upscaling through machine learning, as it is likely to be a vital factor should the company opt to retain a Switch-style form factor while offering greater graphical fidelity in the future. Whether Nintendo will still utilise NVIDIA technology in future devices is also interesting; if it develops its own solution it may not need NVIDIA’s DSSL tools. Of course, depending on what and who you believe, there are reports that ‘4K’ development units are already out in the wild.
Let us know what you think in the comments!
About Thomas Whitehead
Tom has been involved in the games industry for a decade, mostly writing about it or struggling to sell Indie games. Loves reading, writing, hunting Capcom’s fiercest monsters and watching baseball.
Pretty ironic, after what news came of that yesterday.
So yes, 11 devs are working on 4K games then. I wasn’t a believer but this is clear proof.
This will probably fuel even more switch pro rumors.
They will never end, will they?
In my mind this is a dock or links to the dock so it upscales for the TV?
@NintendoKnight I get your point, but not really. This is an interesting tangible bit of information that shows Nintendo is working on its own versions of this technology. Whether you believe the latest rumours about 4K dev kits is up to you but isn’t really linked imo.
TVs upscale lower resolution signals anyway. Upscaled images are – for the most part – jarring and distracting, and I hate them. I’d rather see textures take a hit over resolution or frame rate.
Nintendo should devote more R&D to mastering 1080p rather than pursuing the UHD rat race.
Interesting, if this actually forms into something, that would be cool, but I recommend people to not get their hopes up. Patents don’t always because a real product, so I advise not taking this as immediate proof of a Switch Pro. If anything, it would be a next gen console with backwards compatibility with the Switch.
Unfortunately we may never see Tetris 99 in 4K 😔
@NintendoKnight I haven’t heard about what you are referencing. What news?
Hehe hehe.. N.E.R.D, everytime it makes me chuckle.
This would be a neat feature for when the playing the switch in handheld but at home.
Maybe hyrule warriors would be stable
Yes, but a Switch in handheld mode obviously can’t do that on its own.
And upscaling features on TV sets can introduce latency that interfere with gameplay, which is exactly why Game mode exists.
The examples cited in the article make clear that this is to benefit games that run below the device’s maximum resolution, rather than trying to achieve higher resolutions than the system’s resources are capable of.
In short, this is a way to compensate for games that run at lower internal resolutions, by a very smart method of extrapolating detail.
Give me my pixelated graphics!!! All this fancy schmancy "upscaling"…. no good.
Well no duh! It’s called progress and you can’t stop it as long as people are creating new technology.
@NintendoKnight I was thinking this.
Let’s be honest. There is probably lots of truth to everything Bloomberg has reported on. It’s just a matter of does Nintendo put this system out or drop it. But if third party devs really are being told to develop for it sounds like a done deal.
i feel like "not being 4k" is on the lower end of the hardware limitations when there are quite a few 3rd party ports which have trouble maintaining 720p and a consistent 30fps.
then again quite a few games on ps4 pro have both "performance" and "visual" modes which is something i hope would be common in a theoretical switch revision/successor
@ThomasBW84 Indeed most likely this will be used in the next gen Nintendo console. Also a lot of Nintendo patents never make it into actual hardware.
While it might be true that they are researching on a 4K device (it might not even be a Switch), I guess there’s no point in using a “4K dev kit” while you’re using an upscaler to get 4K.
Maybe this is all about making current gen Switch games compatible with the next generation console.
It’s pretty clear that this tech has nothing to do with the Switch… but that the development of Nintendo’s next system has begun.
(I’m not ready to say goodbye to Switch though 🥺)
@BloodNinja its about nintendo rejects Bloomberg news on the rumored switch pro and all that.
Mochizuki doubters absolutely annihilated once again, when will you guys learn? The guy (and Bloomberg in general) literally has nothing to gain by spreading misinformation, he is just reporting what he has heard from his own reliable sources. Some of which might not even see the light of day. A lot of projects get cancelled internally and we never hear about them.
@meeto_1 “Lots of truth”, where? You probably think Santa is real too?
Nintendo HDS is confirmed ! (Playing both Switch, 3DS and NDS games)
Let’s just get this out of the way now. The Switch Pro just isn’t ever going to happen. Will the NEXT Nintendo system be 4K? Of course. But it just isn’t happening for the Switch.
@Rpg-lover Thank you
@CharlieGirl I think the nature of Switch being a portable console too means that it can exist alongside a successor console just like DS and 3DS did for multiple console generations. 😃 but we’ll probably have updates like DS Lite or New 3DS.
It’s funny to see how people don’t understand what they read. This technology actually means Nintendo is working on something quite unique in terms of processing power, more software based than on hardware.
Both patents explicitly single out Nvidia’s Tensor cores (i.e. the hardware used to run DLSS) as hardware that can accelerate all the necessary matrix multiplication calculations involved in the upscaling algorithm.
Makes sense, as the patents effectively describe how DLSS works.
@Silly_G Well, the introduction text has an example of upscaling from 540p to 1080p, so this is not only limited to 4K. However, if they do actually plan to use machine learning for the task (like Nvidia’s DLSS and Intel’s XeSS do), then the results might not be super impressive when upscaling from lower resolutions.
1: cutting a quote from a single sentence and deliberately leaving out key words like “probably” is very 2021. Well done
2: Santa is 100% real.
I think this is more to upscale games that currently don’t run at 1080p and to run games that do hit 1080p at lower resolutions and upscale them to be 1080p which could then result in better frame rates. Doubt this has anything to do with the Bloomberg article from yesterday.
TV:s just upscales the image, it won’t upscale the graphical resolution of the game itself.
DLSS can make games look like native 4K.
The upscaler in the tv just resize the 720P/1080p image to fit the entire 4K display. It’s not the same thing.
@HotGoomba Hahaha Tetris 99.
@Steel76 Correct my man. If they did anything with this patent its probably already in the new dock! That could be why the ability to update the dock even exists!!
The current problem with DLSS is that it has to be taught every single game it supports, it’s not a switch you can just flip and it starts working in your game.
This could mean Nintendo is researching something different so that developers don’t have to generate petabytes of video from their in-development game to help train the model.
Or if they are going the DLSS route, it could be the devkits are much more powerful than retail units will be so they can pump out realtime 4K visuals to use for training.
I think this is intended for Switch 2.
If they stick with nVidia there’s no doubt in my mind that Switch 2 will have some form of DLSS for docked play.
DLSS can upscale automatically, the devs don’t need to do anything special, so why do they need 4K dev kits?
Btw, Machine Learning can also interpolate frames and turned 30fps to 60fps – high-end TV can already do that but the AI can generate high quality frame rates that looks like native 60fps. So even if the Switch runs a game at 720p @30fps it can be turned into 4K 60fps with Machine Learning.
“Whether Nintendo will still utilise NVIDIA technology in future devices is also interesting; if it develops its own solution it may not need NVIDIA’s DSSL tools.”
Are you joking? Nintendo can’t build ML hardware, Nvidia is still the king in ML hardware. In fact I think Nintendo signed the contract with Nvidia years ago because of their ML expertise, they had the road map planned. They knew the tech they need for a portable to go 4K down the road, it can’t be done by brute force – it needs Machine Learning.
I think we all know the OLED was meant to be a PRO.
Nintendo sent dev kits to a few developers last year. Supply constraints worsened… Nintendo revised approach. Announced an OLED model.
If we still see a Switch Pro, I would be surprised. I think the tech may just be implemented in the next console. Whether its a Super Switch or simply a new piece of hardware.
The theory supports both “Nintendo’s PR” and rumours in my mind. Dev kits usually go out a year+ before consumers get their hands on the retail version.
Not this crap again
Switch Pro confirmed… right?
Hey, what if instead of being for the nonexistent Switch Pro it might actually be technology for their next gen console?
At this point I’d rather have them prepare the Switch successor than a more powerful revision of the Switch.
Like I said the next Switch is going to have 4k/DLSS options but NintenDoomed fandom keep harping gutter trash in hopes they will survive. The form factor will stay and is not going away.
Wasn’t Resident Evil Revelations for the 3DS developed in HD?
I thought it was normal for Devs to develop in Higher Fidelity, then down scale
Switch Pro confirmed. Launching Q2 2022 with exclusive Silksong and Persona 5, NSO swapped for Game Pass. Supports themes, folders and joycon stick autocleaning.
More fuel for the pro rumours, woopeedoooo
@CharlieGirl I honestly can’t see myself being excited for any Nintendo console that’s not a hybrid design.
I find the handheld, motion controls, and VR platforms to have limited appeal, and the home platform already has plenty of choice.
I either need a Switch 2 with full Switch BC, or I am sticking with what I’ve got until I’m completely bored with the content that’s already on the system.
(either that or move to steam deck).
@BulkSlash DLSS 2.0 doesn’t require training specific to each game, that was only on the first generation of DLSS. Implementing DLSS 2.0 is pretty much as simple as making a few simple Nvidia DLSS API function calls, and as long as the hardware is compatible with it, it just works!
@Meteoroid My bad, I thought the fairly recent Digital Foundry video where they swapped the DLSS DLL for Doom Eternal with the one from Rainbow Six was because it had been trained differently. It turns out it was a newer version of DLSS!
@nhSnork Supports themes, folders and joycon stick autocleaning.
That would be welcomed as one can transfer data from one switch to another without using WIFI NSA to do transfer-if it comes to pass.
Obviously a N3DS equivalent for a Switch model is in the works. This was never in question. Upscaling either with Nvidia technology or their own in-house designs is the clear way forward for Nintendo.
The biggest question is always what they choose to do given the current supply shortages for the next few years. How will these unstable times affect a Switch Pro-esque model with 4K upscaling? How will it affect the Switch’s lifecycle projection? Will these next two years delay the next generation console if it delays the Switch Pro rollout?
Nintendo seems to be in a pretty strong position in context with these supply shortages. Sony and Microsoft are fighting over cutting edge chips, while Nintendo can continue pumping out software with its current technology for another year or two at least, given its new launch of the OLED model.
The biggest factor to me was showing BOTW2 in the SwitchOLED promotion. BOTW2 is very likely a winter 2022 launch, and that probably means Nintendo does not plan to release a new model in 2022 if they are promoting a winter 2022 game with the OLED Switch.
If that turns out to be true, then we’re likely looking at a spring~ 2023 rollout of the next Switch model.
My hope is that this, 4K dev kits, and the new dock update feature are related. They want games to be able to be programmed their best, then they’ll update the docks to use AI upscaling to run said content. That lets Nintendo be truthful when they tell their investors they aren’t developing new hardware AND it doesn’t screw the player base
Nintendo has the resources and brilliant technologists at its disposal, you know they’ll come up with something that stays within budgets
If they are patenting it, it feels like they intend to use their own method instead of NVIDIA’s DLSS. So they won’t be bound to NVIDIA chips. Nice to see Nintendo is another company working with machine learning, this has an ridiculous broad application range!
@westman98 This patent is interesting ‘news’. Up until this summer, DLSS was Windows only (https://developer.nvidia.com/blog/nvidia-dlss-sdk-now-available-for-all-developers-with-linux-support-unreal-engine-5-plugin-and-new-customizable-options/). This suggests to me that were testing upscaling themselves with Nvidia’s Xavier SoCs (which do have Tensor cores) back in 2019-2020. And they wrote their own equivalent of DLSS in the absence of non-windows drivers from Nvidia.
Maybe there is substance to the idea that OLED model is the plan B device, when component shortages kicked in and scuppered the Plan A ‘Switch Pro’!
I wonder if Nintendo will stick with their own tech, or use DLSS for the Switch2 (or whatever it’s called, probably not using the name Switch at all)?
@meeto_1 "sounds like a done deal" makes it sound like you’re pretty sure it’s coming. When it’s all crap. The tooth fairy is real as well? Hahaha
I wouldn’t be surprised if this was DLSS at a much smaller scale, as mentioned in this video that games at 540p could be upscaled to 1080p with said technology. It’ll certainly help eliminate some of the issues from many of the more recent games dipping down to questionably low resolutions.
So this means that there were no 4K dev kits right?
Yeah I was thinking that too, frankly that would be a much more interesting application than 4K which I don’t really care about
4k dev kits just sounds absolutely ridiculous considering the costs to make one in the first place. However, there is that possiblity of Nintendo telling developers to make their games "4k-ready" so upscaling would at least be easier. However, like you said, DLSS would be much more interesting than going straight to 4k, which is just… unnecessary
@ModdedInkling DLSS would be much more interesting than going straight to 4k, which is just… unnecessary
Not like they can’t also go 4k but having a 4k/DLSS setup helps their market share grow. True 4K 60fps is only possible on Gaming Desktop/Laptop with SLI/CF setup to achieve this only. There is no other way around this – to get True 4K 60fps in real gaming sense. And only PC systems are the one holding this title. Xvox/Psvox like to claim they are but they are using DLSS for their 4K if those system was to go 4K they would need CF setup since they are using AMD GPU to achieve that but a single AMG GPU isn’t going to do that without lowering resolution alot and the cooling would be massive to handle this. And I don’t think those gamers would pay those massive price to have it. And to benefit this one needs 4K TV to showcase it. So it’s not just the console but the TV has to be 4K not 4K ready to boost this performance.
Now one could’ve said it any better than you. Honestly, it bothers me so much how gamers has been so set on hardware that they completely forgot about the games themselves.
"the techniques herein may be applied to other image sizes (e.g., 720p to 1080p; 480p to 1080p, 1080p to 1440p, 1080p to 4k/3840×2160, 720p to 4k, etc.)"
The patent mentions other resolutions as well, so "DLSS Lite" this is not. The upscaling algorithm remains the same regardless of what the input or output resolution is.
@ModdedInkling The more Graphics options Switch can output the better for gamers.
How about Nintendo figure out stable framerates before they worry about 4K.
Maybe im crazy, but i prefer sharp images where you can see the individual low resolution pixels to blurry, muddy images thare are "upscaled" any day
Tap here to load 63 comments
Leave A Comment
Hold on there, you need to login to post a comment…
Nintendo 64 And Sega Genesis ‘Expansion Pack’ Announced For Switch Online, Launches This October
Pricing, timing and more will be revealed soon
Don’t Worry, Nintendo Isn’t Abandoning Switch Online’s SNES And NES Libraries
People who “don’t upgrade” can still look forward to new additions
It’s Official, Super Nintendo World Is Getting A Donkey Kong Expansion
“Set to open in 2024”
N64 Gems Custom Robo And Its Sequel Are Coming To Nintendo Switch Online In Japan
Playable for anyone with a sub and a JP Nintendo Account
Sin & Punishment For Nintendo 64 Now Has An Official Western Box Art
Thank the Switch Online service
News Metroid Dread Report Volume 9 Outlines Some Tweaks To Key Features
Random Sam Lake Jumped In A Lake To Promote Alan Wake: Remastered
News Join Us For The Nintendo Life Indie Spotlight, Returning On 4th October
News It’s Officially Spooky Season In Animal Crossing: New Horizons
Popular Right Now
Nintendo ‘Blockbuster’ Sale Discounts Hundreds Of Switch …
6 hours ago
Where To Pre-Order Nintendo Switch OLED Model
1 day ago
Nintendo Switch Pro: Everything We Know – 4K Visuals, Nvi…
2 days ago
Best Nintendo 64 Games
2 days ago
Pokémon GO Spotlight Hour Times And Bonus Hour Times: Th…
5 days ago
Random: Wait A Sec, That Sure Looks Like Mario In Sonic T…
1 day ago
Another Fan Claims To Have Reached Nintendo Switch’s Maxi…
13 hours ago
Grand Theft Auto Trilogy Reportedly Heading To Switch
1 day ago
Square Enix Unveils Dungeon Encounters, Directed By Hiroy…
11 hours ago
Patent Reveals Nintendo Is Working On Upscaling Technology
10 hours ago
Review: UnMetal – A Surprisingly Robust Metal Gear Parody…
2 hours ago
Upcoming Nintendo Switch Games And Accessories For Octobe…
5 hours ago
Join 1,218,820 people following Nintendo Life:
© 2021 Nlife Media, partner of ReedPop. Hosted by 44 Bytes.