New Renderer, OSX

[JeroMiya]

Hey guys. Sorry about my hiatus. I\'m currently in California for my Apple internship, which ends in 3 weeks. After which I can begin once again to contribute to the mac version. I haven\'t tried to compile PS in a few months, so I suspect there to be quite a number of new issues. However, one piece of good news I have to report is that CS builds without incident still, and the new renderer actually works, at least on 10.3.4. There are a few glitches, certainly, but it\'s definitely progress from not working at all a few months ago. I haven\'t been able to test the pixel shaders yet, as my big dev machine is at home, and my new laptop doesn\'t arrive till this wednesday. I\'m hoping to see the new renderer in use in CB (other than just using the OR shaders with the new renderer), so I\'m excited that they seem to be working, although I can\'t imagine the performance implications of real time lighting and shadowing for a large scale MMORPG, but of course EQ2 gives one hope!

Has anyone tried to compile PS recently?

[dfryer]

Yep, and we just fixed a major bug that showed up in the MacOS X build due to un-cross-platform-compatible assumptions about the order of execution of parameters.

As far as I know, PS doesn\'t yet use any fancy shaders, just the OR emulation (lightmapping).  

Right now I\'m twiddling with VFS so that I can make a self-contained PS install (for the mac) that doesn\'t require any environment variable muckery.  I\'ve also found that disk images can include license agreements (which the PS team requires) so we can get away from installer scripts entirely (e.g. I would like to achieve drag-n-drop install)

Only problem with NR on 10.2 that I\'ve experienced is random polygons (not part of world geometry) flickering in and out of existance when plane clipping is used instead of stencil, but I think it\'s currently less of a problem than it used to be.  Glad to know it\'s good on 10.3 as well.

SmokeD (hangs out on #crystalspace and #planeshift) also is building on Mac OS X - he\'s put together a compile-from-scratch guide, complete with installs of necessary dependencies.  I think it\'s for 10.3 - you might want to talk with him and test it out.  I think it\'s fair to say that we can support running PS on 10.2.8, and restrict the compilation instructions to 10.3 and up.

How\'s Apple?

EDIT: btw, I\'ve never been able to get shaders using ARB_vertex_program working correctly, even though my drivers *supposedly* support it via emulation.  It\'d be nice if this is a 10.2-only issue.

2ND EDIT:  Another thing that both of us should do is profile the new renderer under Mac OS on our fairly different machines  The new \"csbench\" application is designed for putting the NR through its paces, so that would be a good place to start.

[JeroMiya]

Well, my new Powerbook comes in Wednesday, and my desktop only supports vertex programs, so I can only test those shaders that use only vp. However, here is a quick run (debug compile. Also, notice the invalid reporting of pixel format depths, and the still remaining apparently reference count related errors):

Code: [Select]


CS] jeremybe% ./csbench.app/Contents/MacOS/csbench -verbose -recalc -relight
Crystal Space for MacOS/X 0.99 r0 [MacOS/X-powerpc-GCC]
Ported to MacOS/X by Eric Sunshine


crystalspace.graphics3d.opengl:
  Texture LOD bias -.3

crystalspace.canvas.glosx:
  MacOS/X 2D OpenGL driver for Crystal Space 0.99 r0
Written by Matt Reda
 

crystalspace.canvas.openglcommon:
  OpenGL renderer: NVIDIA GeForce2 MX OpenGL Engine (vendor: NVIDIA
  Corporation) version 1.1 NVIDIA-1.3.18
  Using windowed mode at resolution 640x480.
  Pixel format: Color: 0 Alpha: 0 Depth: 0 Stencil: 0 AccumColor: 0
  AccumAlpha: 0 MultiSamples: 0

crystalspace.canvas.opengl.extmgr:
  GL Extension \'GL_ARB_multitexture\' found and used.
  GL Extension \'GL_ARB_texture_env_combine\' found and used.
  GL Extension \'GL_ARB_multisample\' not found.
  GL Extension \'GL_version_1_2\' found and used.
  GL Extension \'GL_ARB_texture_cube_map\' found and used.
  GL Extension \'GL_EXT_texture3D\' not found.
  GL Extension \'GL_ARB_vertex_buffer_object\' found and used.
  GL Extension \'GL_SGIS_generate_mipmap\' found and used.
  GL Extension \'GL_EXT_texture_filter_anisotropic\' found and used.
  GL Extension \'GL_EXT_texture_lod_bias\' found and used.
  GL Extension \'GL_EXT_stencil_wrap\' found and used.
  GL Extension \'GL_EXT_stencil_two_side\' not found.
  GL Extension \'GL_ARB_point_parameters\' found and used.
  GL Extension \'GL_ARB_point_sprite\' not found.
  GL Extension \'GL_ATI_separate_stencil\' not found.

crystalspace.graphics3d.opengl:
  Maximum texture size is 2048x2048
  Point sprites are NOT supported.
  VBO is supported.
  Stencil clipping is used for objects >= 50 triangles.

crystalspace.graphics3d.shadermgr:
  Loaded compilerplugin crystalspace.graphics3d.shadercompiler.xmlshader,
  compiler: XMLShader

crystalspace.canvas.opengl.extmgr:
  GL Extension \'GL_ARB_texture_env_dot3\' found and used.

crystalspace.graphics3d.shader.fixed:
  Multitexture units: puny 2

crystalspace.graphics3d.shadercompiler.xmlshader:
  Shader \'or_lighting\': Technique with priority 100 succeeds!

crystalspace.mesh.object.thing:
  Lightmap quality=3


crystalspace.engine.notify:
  Lightmaps are not up to date (no \'lm_precalc_info\' found in cache).
  Recalculation of lightmaps forced.
  Initializing lighting (3 meshes).
  Shining lights (3 lights).
Doing light 0
Doing light 1
Doing light 2
  Time taken: 1.3650 seconds.

csbench:
  ================================================================
  Crystal Space compiled in debug mode.
  Compiler \'GCC\', platform \'MacOS/X\', processor \'powerpc\'
  ================================================================
  Benchmark stencilclip (Stencil clipping is used)...
  PERF:stencilclip:115:2:57.5: (115 frames in 2 seconds: 57.5 fps)
  ================================================================
  Benchmark planeclip (glClipPlane clipping is used)...
  PERF:planeclip:117:2:58.5: (117 frames in 2 seconds: 58.5 fps)

crystalspace.graphics3d.shadercompiler.xmlshader:
  Shader \'or_lighting\': Technique with priority 100 succeeds!

csbench:
  ================================================================
  Benchmark or_lighting_100 (Shader or_lighting with priority 100)...
  PERF:or_lighting_100:119:2:59.5: (119 frames in 2 seconds: 59.5 fps)

crystalspace.graphics3d.shadercompiler.xmlshader:
  Shader \'or_lighting\': Technique with priority 75 fails. Reason: vertex
  program failed to load.
  No technique validated for shader \'or_lighting\'
  Shader \'or_lighting\': Technique with priority 50 succeeds!

csbench:
  ================================================================
  Benchmark or_lighting_50 (Shader or_lighting with priority 50)...
  PERF:or_lighting_50:87:2:43.5: (87 frames in 2 seconds: 43.5 fps)

crystalspace.graphics3d.shadercompiler.xmlshader:
  Shader \'ambient\': Technique with priority 50 succeeds!

crystalspace.canvas.opengl.extmgr:
  GL Extension \'GL_ARB_vertex_program\' found and used.
  GL Extension \'GL_ARB_fragment_program\' not found.

crystalspace.graphics3d.shadercompiler.xmlshader:
  Shader \'light_bumpmap\': Technique with priority 350 fails. Reason: fragment
  program failed to load.
  No technique validated for shader \'light_bumpmap\'
  Shader \'light_bumpmap\': Technique with priority 300 fails. Reason: fragment
  program failed to load.
  No technique validated for shader \'light_bumpmap\'
  Shader \'light_bumpmap\': Technique with priority 250 succeeds!

csbench:
  ================================================================
  Benchmark light_bumpmap_250 (Shader light_bumpmap with priority 250)...
  PERF:light_bumpmap_250:12:2:6: (12 frames in 2 seconds: 6 fps)

crystalspace.graphics3d.shadercompiler.xmlshader:
  Shader \'light_bumpmap\': Technique with priority 200 fails. Reason: fragment
  program failed to load.
  No technique validated for shader \'light_bumpmap\'
  Shader \'light_bumpmap\': Technique with priority 150 succeeds!

csbench:
  ================================================================
  Benchmark light_bumpmap_150 (Shader light_bumpmap with priority 150)...
  PERF:light_bumpmap_150:14:2:7: (14 frames in 2 seconds: 7 fps)

crystalspace.graphics3d.shadercompiler.xmlshader:
  Shader \'light_bumpmap\': Technique with priority 100 succeeds!

csbench:
  ================================================================
  Benchmark light_bumpmap_100 (Shader light_bumpmap with priority 100)...
  PERF:light_bumpmap_100:20:2:10: (20 frames in 2 seconds: 10 fps)

crystalspace.graphics3d.shadercompiler.xmlshader:
  Shader \'light_bumpmap\': Technique with priority 50 succeeds!

csbench:
  ================================================================
  Benchmark light_bumpmap_50 (Shader light_bumpmap with priority 50)...
  PERF:light_bumpmap_50:51:2:25.5: (51 frames in 2 seconds: 25.5 fps)

crystalspace.pluginmgr.loadplugin:
  WARNING: could not load plugin \'crystalspace.graphics3d.shader.soft\'

crystalspace.graphics3d.shadercompiler.xmlshader:
  Couldn\'t retrieve shader plugin \'crystalspace.graphics3d.shader.soft\' for
  in shader \'light_bumpmap\'
  Shader \'light_bumpmap\': Technique with priority 0 fails. Reason: vertex
  program failed to load.
  No technique validated for shader \'light_bumpmap\'
SCF WARNING: 1 unreleased instances of class crystalspace.console.output.simple!
SCF WARNING: 1 unreleased instances of class crystalspace.font.server.default!
SCF WARNING: 1 unreleased instances of class crystalspace.graphics2d.glosx!
SCF WARNING: 1 unreleased instances of class crystalspace.graphics3d.opengl!
SCF WARNING: 1 unreleased instances of class crystalspace.graphics3d.shadercompiler.xmlshader!
SCF WARNING: 1 unreleased instances of class crystalspace.graphics3d.shadermanager!
SCF WARNING: 1 unreleased instances of class crystalspace.syntax.loader.service.text!
SCF WARNING: 1 unreleased instances of class crystalspace.utilities.reporter!
SCF WARNING: 1 unreleased instances of class crystalspace.utilities.stdrep!


[dfryer]

Looks like VBO support was added to the drivers between 10.2 and 10.3, since we both seem to have Geforce2MXes. What speed is your CPU running at?

BTW, it was the bug that we just fixed that caused the strange slowdowns when the server spawned the world - all the z and x coordinates were swapped by the client on initial load, so they were \"off\" the edge of the world..  The resulting falling caused a slowdown as the collision detection method took more CPU the faster the object was moving.. this was what was causing the trouble.

Do you have any experience with the \"opengl profiler\" that Apple supplies?  Perhaps we could use that to identify mac-specific bottlenecks.  I suspect (from your and my csbench scores) that many things are being handled in software when it comes to the geforce2MX.  CS now has a \"driver database\" xml file which allows disabling of \"available\" features, to increase speed.  We also might want to look into twiddling that for low-end cards on the Mac.

Just a few ideas
Thanks,
Daniel

[JeroMiya]

Yes, however for detecting software fallbacks you might want to also take a look at OpenGL driver monitor. I have less experience with this program but I have used it before. It\'s pretty straightforward.

So, the mobs were falling off world? That\'s interesting, however, it seems a little odd that they would be causing increasing CPU usage, especially when they are not going in the same direction as the world. I would suggest two things. First, implement a terminal velocity. Second, for each area, have a cutoff plane that /spawns anything that falls down past that point and reports/logs it.

10.2.8 is a good min target. Halo runs on 10.2.8. Looking forward a few months, however, we may want to target 10.3.4 as the min target as many  serious bugs have been fixed in this version.


Any word on ETA for fixing the ref count errors as seen in that log? What\'s going on there?

-Jeremy Bell

[dfryer]

As far as I understand it, those refcount errors are a Very Low Priority, as it\'s (apparently) not a problem that ever manifests itself in a loop, i.e. it doesn\'t result in a true memory leak, and is related to the order in which things are destroyed.

Now that the coordinate swapping bug is fixed, I don\'t think a terminal velocity/fall plane is needed.  It was just one of those weird things that is unlikely to be caused by anything else - or if it is, it\'ll be fairly obvious.

I\'m going to be sticking with 10.2.8 for a while, as I don\'t have budget for anything else.  I did use the NR walktest to walk around Molecular Blue Hydlaa (with lightmapping sometimes disabled, as I couldn\'t get such an old map to relight correctly) and I was getting 10-40 fps, unless I spun around really quickly.  I consider that quite reasonable considering that a year and a half ago, we would have been lucky to get 1 fps.  

I\'ll try to get a working arrangement of plugins and config files suitable for an installer, and let you know the details.  Hopefully it won\'t require any patching of VFS or anything drastic like that, but IMO CS should be designed to do drag and drop installs on the Mac.

[dfryer]

Okay, I can get a double click launch in OS X, although it\'s failing on some csarray assertions eventually, which I\'ll have to look into.  VFS settings in the PS code need to be slightly modified in order for it to work; specifically wherever $.$/ (the current directory) is used, $^ (the directory that the executable (on win/linux) or the .app (on OS X) lives in) needs to be used instead.  A modified vfs.cfg is also helpful if we want the CS install to be hidden inside the .app wrapper.