asynchronous ressource loader/manager for games
Posted: Tue Aug 18, 2009 11:02 pm
Finaly i release my ressource manager on the international PB community.
What is it? What can it do? And why should you use it?
Well it's .dll that simplifies and speeds up the handling of ressources on games. With ressources i mean any kind of content data, like graphics, sounds and stuff. Well it's no problem to load that ressources just with PB, right. But my ressource manager have some interessting feature.
The core feature is, that it loads and decompresses the ressources asynchronously. That means you request a ressource and the manager loads it and decompresses it on 2 separate threads while your program it not suspended. That makes it very very easy for you to load level data while the player is playing the level. This is known as level streaming or world streaming. You can make very very big levels with low memory usage. Or make a game without any loading times at all on the actual game play.
Actualy i developed this ressource manager for exacly that reason and i currently using it for level streaming.
If you want to program level streaming you need a high performance loader. So i did very much effort into optimization. And that payed out. Using my ressource loader to load files is much faster than loading them directly from hard disk. And here is why: If you request more than 1 ressource at a time, it will load from hd and decompress at the same time to get rid of the HD delay. While the second ressource is loading from HD the first ressource is decompressing. The access to the files is also incredible fast. The manager indexes the ressources if you initialize it. That means it gets all the informations about the ressources and from were and how to load them befor you actualy request them. Therefor it can access them realy fast. And of course it uses a hash table for indexing.
It even comes with it's own file format for a ressource container or ressource archive. How ever you want to call it. This format is designt for fast access and to support 2 more features.
The first feature is: multilingual ressources
A ressource can have 255 different language versions all packet into the same container. You only have to worry about the language on creation of the ressource container files. After that the ressource manager takes care about the language stuff. The principle is easy. You asign a number to every languageversion of a ressource. That number is representing a language. If you initialize the manager, you pass a number to it and it will only index the languageversions with the matching number. If a ressource have no matching, it indexes the first language version of it thats in the container.
The second feature is: ressource versioning
You can assign versionnumbers to ressources. The manager will only index the highest version of a ressource.
Let's say you want to make a patch for your game. You don't have to change the whole container file. You just make a second container file and pack a ressources with higher version numbers in it. They will automaticly replace the older versions.
Thats introducing another feature. The ressource loader can handle multiple ressource container files. In fact it does not care how many or how you name them. On initialization you pass a path to the manager. It will index all ressource container files, that are on that path. And they are indexed as it would be just one container. You don't need to worry about in which container you places which ressource. The manager will take care about that. You just request the ressources you need by there name.
Well, i hope this long text don't bored you. And i hope someone can need this ressource manager.
It comes in 32bit and 64bit DLL's only for Windows.
At the moment it supports JCalG1 and ZLib for compression. Both of them on 32bit and 64bit. JCalG1 even up to 25% faster on decompression than the PB integrated decompressor.
Well i could write on for hours. But i think thats enough. If you have questions just ask them. The download contains a example code and a tool to build ressource container.
More compressions are planed to implement. if you know about a free high performance compression you want to see on my manager, just contact me.
Download: Thorium Ressource Manager v0.02
It's not the final version but it is stable.
It's free to use for non commercial and commercial projects.
Edit:
One thing: There are some more DLL's included in the package.
ThorRes32_JCalG1.dll
ThorRes64_JCalG1.dll
ThorRes32_ZLib.dll
ThorRes64_ZLib.dll
That are plugins for the ressource manager.
I decided to implement different compressions with a plugin system mainly to reduce the size of the manager. You just need the plugins with the compressions you want to use. If your project don't uses all compressions you don't have to include the unused plugins in your installation. Besides this, a plugin system makes it much more easy for me to add more compressions.
What is it? What can it do? And why should you use it?
Well it's .dll that simplifies and speeds up the handling of ressources on games. With ressources i mean any kind of content data, like graphics, sounds and stuff. Well it's no problem to load that ressources just with PB, right. But my ressource manager have some interessting feature.
The core feature is, that it loads and decompresses the ressources asynchronously. That means you request a ressource and the manager loads it and decompresses it on 2 separate threads while your program it not suspended. That makes it very very easy for you to load level data while the player is playing the level. This is known as level streaming or world streaming. You can make very very big levels with low memory usage. Or make a game without any loading times at all on the actual game play.
Actualy i developed this ressource manager for exacly that reason and i currently using it for level streaming.
If you want to program level streaming you need a high performance loader. So i did very much effort into optimization. And that payed out. Using my ressource loader to load files is much faster than loading them directly from hard disk. And here is why: If you request more than 1 ressource at a time, it will load from hd and decompress at the same time to get rid of the HD delay. While the second ressource is loading from HD the first ressource is decompressing. The access to the files is also incredible fast. The manager indexes the ressources if you initialize it. That means it gets all the informations about the ressources and from were and how to load them befor you actualy request them. Therefor it can access them realy fast. And of course it uses a hash table for indexing.
It even comes with it's own file format for a ressource container or ressource archive. How ever you want to call it. This format is designt for fast access and to support 2 more features.
The first feature is: multilingual ressources
A ressource can have 255 different language versions all packet into the same container. You only have to worry about the language on creation of the ressource container files. After that the ressource manager takes care about the language stuff. The principle is easy. You asign a number to every languageversion of a ressource. That number is representing a language. If you initialize the manager, you pass a number to it and it will only index the languageversions with the matching number. If a ressource have no matching, it indexes the first language version of it thats in the container.
The second feature is: ressource versioning
You can assign versionnumbers to ressources. The manager will only index the highest version of a ressource.
Let's say you want to make a patch for your game. You don't have to change the whole container file. You just make a second container file and pack a ressources with higher version numbers in it. They will automaticly replace the older versions.
Thats introducing another feature. The ressource loader can handle multiple ressource container files. In fact it does not care how many or how you name them. On initialization you pass a path to the manager. It will index all ressource container files, that are on that path. And they are indexed as it would be just one container. You don't need to worry about in which container you places which ressource. The manager will take care about that. You just request the ressources you need by there name.
Well, i hope this long text don't bored you. And i hope someone can need this ressource manager.
It comes in 32bit and 64bit DLL's only for Windows.
At the moment it supports JCalG1 and ZLib for compression. Both of them on 32bit and 64bit. JCalG1 even up to 25% faster on decompression than the PB integrated decompressor.
Well i could write on for hours. But i think thats enough. If you have questions just ask them. The download contains a example code and a tool to build ressource container.
More compressions are planed to implement. if you know about a free high performance compression you want to see on my manager, just contact me.
Download: Thorium Ressource Manager v0.02
It's not the final version but it is stable.
It's free to use for non commercial and commercial projects.
Edit:
One thing: There are some more DLL's included in the package.
ThorRes32_JCalG1.dll
ThorRes64_JCalG1.dll
ThorRes32_ZLib.dll
ThorRes64_ZLib.dll
That are plugins for the ressource manager.
I decided to implement different compressions with a plugin system mainly to reduce the size of the manager. You just need the plugins with the compressions you want to use. If your project don't uses all compressions you don't have to include the unused plugins in your installation. Besides this, a plugin system makes it much more easy for me to add more compressions.