Author Topic: Save shading information with file  (Read 359 times)

Dr PR

  • Hero Member
  • *****
  • Posts: 5440
Save shading information with file
« on: March 11, 2017, 11:21:47 AM »
This really isn't a new request, but it is worth repeating.

When I open a file and then shade it the shading operation takes a relatively long time. For example, I just loaded a 59.8 Mbyte drawing and it took about 10 seconds. But the initial OpenGL Gouraud shading took about two minutes - about ten times as long as to open the file. I have files as large as 750 Mbytes, and the initial shading of these files can take over an hour (one file takes 90 minutes)! It all depends upon how many renderable objects (solids, planes, grids) are in the files, and large files can have nearly a million objects. When I load one of these files I have to wait an hour or more before I can start working again. PITA!

I say the "initial" rendering because after this first shading new shading operations take only a few seconds at most, even in the largest drawings. So obviously it is the initial calculations of normals and such that is taking the greatest part of the time.

All of this initial delay could be avoided if we had the option of saving the shading information was saved with the drawing. Since this information is available when the file is saved, it should be fairly simple to save the information. The only penalty it would cause if a somewhat longer save operation and greater file size.

I gave worked with files larger than a gigabyte in Photoshop. You do notice a longer load/save time with a gigabyte file - but it is just a few tens of seconds at most - not a hour or more.

I have a one terabyte drive primarily dedicated to DesignCAD files. But since all of my DesignCAD files that have accumulated in nearly 30 years of using the program (yes, I still have everything dating back to 1987) require only 203 gigabytes of storage, I have plenty of reserve space on the hard drive. So DesignCAD file sizes are not a problem.

So what "penalty" would we see if we had the option to save shading information? It is basically a matter of how much information must be saved. So I did an experiment. I used the Microsoft Resource Monitor program to determine how much memory DesignCAD V26.0 uses. The relevant numbers are the size of the "Working Set" - the amount of physical memory currently in use by the process.

DesignCAD V26.0 started with a blank file = 69.7 Mbytes.
Open a 59.8 Mbyte drawing = 267.8 Mbytes - an increase of 198.1 Mbytes.
Shade with OpenGL Gouraud = 329.4 Mbytes - an additional increase of 61.6 Mbytes.
Sagde with GDI Phong with shadows = 330.2 Mbytes = an increase of 0.8 Mbytes.

Shading increases the working information by about 62 Mbytes. If this was saved with the drawing the file would be about 121 Mbytes, or about double in size. File load time would be about 20 seconds.

It may be that some of the increase would not have to be saved. And it might be necessary to save some of the initial 198 Mbyte increase when the drawing was first loaded. But if all drawing related information (198.1 + 61.6 = 259.7 Mbytes was saved along with the drawing the file would be about 320 Mbytes. This would be a 5X increase of file size. So what?

OK, so it would take about 50 seconds to load the drawing. But, in theory, this would save a couple of minutes waiting for the drawing to shade. But for really large drawings the improvement could be much greater. A 750 Mbyte drawing would grow to 3.75 Gbytes. That might take 10 Minutes to load - but that would be a small price to pay to not have to wait an additional hour and a half to be able to start working!

My 1 Terrabyte hard drive can store over 250 3.75 Gbyte drawings. Since it has taken me over ten years to produce the 750 Mbyte drawing, I probably won't live long enough to create another 250 such drawings!

****

For those who work with very large 3D drawings the ability so save the shading information would be a big performance boost.

***

What about backward compatibility? Older versions of DesignCAD will not know what to do with the additional information. But this is already true for many of the features added to the newer versions.

Some programs insert an "end of file" mark at the end of the data part of a file (because disk sectors store information in segments not related to file size, and the last bit of data read from a sector may be meaningless). If so, it would be possible to add a "shading information follows" mark immediately after the "end of file mark." Older versions will stop reading at the "end of file mark," but later versions could look for the "shading information follows" mark and continue reading until the next "end of file mark" comes along.

In any case, backward compatibility isn't a real problem if you save the file without the shading option, or save it in an earlier DesignCAD version format.

Phil
DesignCAD user since 1987

bdeck

  • Hero Member
  • *****
  • Posts: 815
Re: Save shading information with file
« Reply #1 on: March 19, 2017, 12:09:39 PM »
Hi Dr PR,

The option to save the entire 330meg memory image (from your first example) intact seems to make the most sense. Should cut the load/startup time by a couple orders of magnitude, with less coding.

bd


« Last Edit: March 19, 2017, 12:33:02 PM by bdeck »

Dr PR

  • Hero Member
  • *****
  • Posts: 5440
Re: Save shading information with file
« Reply #2 on: March 19, 2017, 08:13:58 PM »
bd,

I did think of that. Back in the '70s a friend who hacked together his first computer (back before some idiot journalist gave hacking a bad name) wrote a simple operating system that just saved the entire RAM contents to disk when the computer shut down, and reloaded it when it started up again. You just resumed operations where you left off.

Then in the 80s and 90s I used the DOS-based Tango PCB program for circuit board design - by far and away the best circuit board layout program ever written. It was created by people with actual circuit board design. Every time you saved a file the entire state of the design was saved. When you reloaded the file your cursor was in the exact same position it had been when you saved it - all options returned to where they were. It was as if you had never stopped working on the design.

However, to do this every variable in the the internal state of the program and the CPU must be saved with the file. The existing stack(s) must be saved along with all CPU memory allocation variables and pointers, etc. Leave out one value and things can get messed up. If the programmers are smart, before the file is saved they will put the CPU internal registers into a known "shutdown" state, and reset the CPU to this state before opening a file. But with multiple CPU processors and  complex memory paging it might be very difficult to do. You really need to save the entire state of the operating system. But reloading this later will screw up other programs that might be running in the background.

In modern operating systems physical memory is allocated by the OS to programs on a first-come first-served basis. So every time a drawing is opened it may be loaded into RAM at an entirely new physical address. All program variables must be adjusted to comply with this new arrangement. Even though the program sees "virtual" addresses and doesn't actually know what the physical addresses are, there is no guarantee that "relative" addresses will be the same within blocks of virtual memory. You can't just dump a memory image back where it came from like my friend did 40 years ago.

I think just saving a memory image won't work.

****

The program will need to save all variables relative to the drawing, including shading information, in an organized way so they can be restored when the drawing is opened again. It already does this for many drawing variables. But you can't just write over existing program variables when you load a drawing because other drawings might already be opened. Also, the state of communications between the program and operating system must not be changed. So the program has to read in the variables and ask the operating system to allocate memory and then put the variables wherever the OS says it is OK to put them.

But loading data from the file really isn't any different as far as memory allocation goes than it is when the program generates all the shading information after loading the drawing. The program still has to ask the OS for memory. But reading the variables from the file will eliminate the lengthy recalculation and just require getting a place to store them from the OS.

I think. DT and Company will understand this much better than I do.

Phil
DesignCAD user since 1987

Rob S

  • Hero Member
  • *****
  • Posts: 4332
    • Construction Estimating Program for General Contractors
Re: Save shading information with file
« Reply #3 on: March 21, 2017, 07:03:06 AM »
Just use windows sleep mode, it seems to work amazingly well!!

This will not help with designcad itself needing to close periodically to clean itself up and restart. 
User since Pro-design

Dr PR

  • Hero Member
  • *****
  • Posts: 5440
Re: Save shading information with file
« Reply #4 on: March 21, 2017, 09:08:34 AM »
Rob,

That won't help at all. I have thousands of drawings and they are not all open at the same time.

Occasionally I want to open a large drawing that may not have been used for months or years. It is these drawings that I wish to open faster, without having to wait up to an hour and a half for the initial shading to be complete.

Opening the file in wireframe view - by ignoring shading settings - is not an answer. With complex 3D drawings where the wireframe view is a maze of crossing lines it is essential to be able to see a shaded versions in order to determine just what you are looking at.

Phil
DesignCAD user since 1987

DrollTroll

  • Kindly Curmudgeon
  • Administrator
  • *****
  • Posts: 4152
Re: Save shading information with file
« Reply #5 on: March 21, 2017, 10:48:07 AM »
If we do this, most likely we'd have to have a separate stream in the file dedicated just to the rendering info.
25 years with DesignCAD

Dr PR

  • Hero Member
  • *****
  • Posts: 5440
Re: Save shading information with file
« Reply #6 on: March 21, 2017, 02:31:30 PM »
That is what I was thinking. Save the drawing normally, and then save the shading information in such a way as the shading operation could retrieve it without doing all of the calculations.

There are several decisions to make.

1. The option to save and load shading information should be included on the"Qptions/Options" dialogs and should remain in effect for all files until changed to set a common behavior for all files. For files that do not include shading information this option would be ignored.

The default should be to not save/load shading information so the program works as it always had. But when the general save/load shading information flag is set it should remain in effect every time the program starts to continue operations as they were the last time the program was closed.

2. The "File/Save" could also have an option to override the general "Options/Options" settings for each file.

3. Even though a drawing may be viewed in wireframe at the time it is saved, the current shading information should be saved with it if the general option is set (from 1 above) or if there is an option in the Save dialog (2 above) to save shading information with the file is selected.

4. The "File/Open" function should also have a local option to override the general setting. It would allow just the drawing to be opened without loading previous shading information. This would speed up loading of large files.

Maybe a dialog should open saying the drawing has saved shading information and asking if it should be loaded from the file IFF the general save/load shading information is not set to save/load shading information.

This should be separate from the "Ignore view shading flags" option. This is a very useful option for large drawings that were saved while shaded. Opening in wireframe really speeds up the file open operation. You want this option to still be available for loading large files that were saved when shaded but did not include shading information.

Phil

DesignCAD user since 1987