More agile development process

[redhound]

Thanks for detailed explanation, Eredin! 

About repository separation again - the high degree of sharing between various applications could be resolved by creating a set of "core" libraries. In perspective, it could provide clean internal API and better testablity, IMHO.

But even in the current state, some stuff may be moved to separate repos: website, forum (well, maybe, just forum theme), default skins, public web art assets (think about something like "PS Fan Kit"). Definitely, there are more PS players with some web development skills, than with C++/CrystalSpace ones.

Also, it could be very helpful to have a public list of concrete missing art assets of the game, like: "We need a 3D model for two-handed sword. We need a texture of red wood." - plus concrete tech requirements. The bug tracker can be used to organize that.

[redhound]

This would be very nice. I have begun referring to the flyspray issue number associated with the bug in commit notes. It's a poor substitute. Do you know of anything that integrates with SVN ? Better yet, would you be interested in investigating the possibility of tying our current bug track to our current sourc control system?

AFAIK, FlySpray do not provides VCS integration out-of-the-box, even its latest version. But some others do, like TheBugGenie.

[LigH]

Also, it could be very helpful to have a public list of concrete missing art assets of the game, like: "We need a 3D model for two-handed sword. We need a texture of red wood." - plus concrete tech requirements. The bug tracker can be used to organize that.

Certainly helpful, but not that simple. Years ago, a big flaw in the organization of different departments was the lack of "inter-disciplinary communication". One example: A task to make an icon for an item called "Ruby Ring". Okay, make a metal ring containing a ruby ... wrong. Later in the quest I got to know that the ring was cut out of a huge ruby crystal. That looks completely different, of course. So before an prospect artist gets the task to make a model or texture, first the different departments need to collect enough base knowledge to enable the artist to understand enough of the context, without e.g. spoiling a whole quest where this item may be involved. The ABC license protects the game content as a whole. It is hard to release too many details before they are meant to be known in public...

[redhound]

Well, discommunication happens. Although, from your description, LigH, I understand how that "Ruby Ring" should look like - yet you don't spoiled anything. I think in most cases it is just about clear specifications, don't you agree?

I understand what closed art license is greatly influencing current development process, but... anyone who don't care about licenses could just open PlaneShift art and data dirs from released client and re-use the contents for any gods-known purpose. So at least client-side art assets are public (although, still under ABC). Most server-side text art (books and quests) could be "stolen" relatively easy (book could be saved to text file, NPC quest dialogs could be taken from logs, etc.) So I think about those assets as of public too.

On the basis of that, why don't make certain pieces of art public to allow developers to work on them openly? It's not about making another game on the PS base, nor I see anything about spoiling game content in ABC license (it's about Game policy 11).

[Serneum]

The biggest question is, given the size of our team, would we get enough benefit to make it worth the time and effort to convert ?

With our current approach each dev/prospect/contributor essentially takes an informal branch to build their local environment then the changes are merged back in manually by the reviewing dev. I'm not sure how formal branching would change things and my experiences with anything other than manual merges have been disastrous.

You can actually convert from SVN to Git pretty easily nowadays. I believe Git can just check out the SVN repo and go from there. We migrated 100s of repos to Git at work and it didn't take too long.

One final consideration, admittedly minor : we get a lot more project starts than completions. If we did a formal branch for each project we'd have a great many dead branches. I guess that's not really an issue for any source control system. It just feels untidy to me.

You could have devs do local branches for their features/fixes. The branches are only on their machines until they get pushed up to the Git repo. You could also handle this the same way we do at work where you create a feature branch locally and just push to any branch of your choice.

Considering the high degree of sharing betweent the client, server and npcserver...how would we accomplish this without creating more work for ourselves?

You use some tool that has dependency management, like Maven, and have it pull down dependencies as part of the build. You can also modify those projects locally, push a locally modified version to a local Maven, and then the local builds for Planeshift would pull down the new version

This would be very nice. I have begun referring to the flyspray issue number associated with the bug in commit notes. It's a poor substitute. Do you know of anything that integrates with SVN ? Better yet, would you be interested in investigating the possibility of tying our current bug track to our current sourc control system?

I'm not sure of anything that integrates well with SVN. Honestly, I'm not sure of what integrates well with Git off the top of my head. I've used Pivotal and RTC. Of the two, I prefer Pivotal, but the only way to tie it together with the SCM system was through a build tool called AnthillPro and it's not free. A free alternative would be something like Jenkins, but you've already discussed automated build systems

Regarding unit testing, I've done some in java with junit and found that while very useful, you end up writing tests for each and every method/function and the tests are only as good as you make them. Ie. A test might test basic functionality (does it compute for one set of parameters) or the range and scope of the function (does it work for any combination of parameters including zeros nulls max and min values). I'm curious about whether or not there is a better way of doing this in c/c++ or if this is just the price to pay for added assurance.

I'm pretty sure what you describe with unit testing is exactly how it should be. Your methods should be easily testable to prove out that they do whatever minor function they have been designed to handle. You could add in libraries like Mockito to mock out full objects and test behavior based on mocked input/fields. You want as much code coverage as you can get and tests should be short and simple, whenever possible

All of this being said, it sounds like the team is a lot smaller than I expected and it's probably not worth the time and effort to change things right now, but it's something that could be considered if more people got on board.

Sorry my reply was so late. I stopped getting emails about replies for some reason

[Lucubro]

I am going to try to answer to this point by point by expanding the excellent considerations from Eonwind, Eredin and Serneum. I tend to write too much and this is a thread with a broad topic so I apologize in advance for the wall of text .

1) In general, I agree with Eredin on this one. Git is great to implement the feature branch workflow, but given the size of the team and the fact that each developer works on a single feature, this wouldn't provide a great benefit right now. But it would make sense to switch to git if we decided to completely change platform and migrate the code to github (as suggested by redhound). Github has the advantage of making the "fork and pull request" workflow very easy which IMHO fits very well the current PS development process and adds few advantages:

• It's clean: instead of creating a lot of branches in the main repo, the prospect/anybody can create a copy of the repository in its own github account, and if he drops the task he doesn't leave dead branches anywhere.
• The pull request is basically a work-in-progress patch. Github creates an open discussion associated with each pull request and anybody is able to check the current changes in the code and suggests modifications to add to the pull requests on-the-fly. Once everything is ready, a member of the official team can merge the pull request into the main repo.
• The main repo remains clean. Each feature requires several commits, and (especially for complicated features) there can be commits that introduce bugs and break the build of the main repo. The pull request is merged only at the end, so this possibility is greatly reduced.
• The project would never miss a bit of a contribution. Working on its own online repo, a prospect can make as many commits as he wants instead of creating big monolitic patches. If he drops the task, those small contributions are not lost because github creates a local copy when a pull request is submitted, so they can be merged and picked up by someone else.

Moreover

• On github there is a fully-featured, easy-to-use issue tracker integrated with the repository.
• I don't have numbers, but I have been involved in several projects using svn and git, on sourceforge and github, and some developers tend to stay away from svn simply because they don't like it. Moreover, I have the feeling that the developers community is gradually moving to git and github in particular, so all the "kool kids" hang out there .

To sum up, I think moving to git not only would speed up and simplify the workflow, but also that it would be better suited to attract new developers (which is IMHO the killer advantage considering how much lack of "manpower" is slowing down PS ).

Ah, and as Serneum pointed out, moving an existing svn repo to git is very easy https://help.github.com/articles/importing-from-subversion/.

2) I agree with Eonwind on this one. There are many cases in which the same feature in order to be implemented touches multiple parts of the code: client, server and common libraries. Splitting them could make it difficult to track the changes in the history.

3) Everybody agrees on this one .

4) Unit testing is great and I loooove test-driven development  . This being said, doing decently comprehensive testing in complex videogames is generally tricky. All the existing PS code has beed tested for many years in-game, so I'm not so sure that all the hours needed to write all the unit tests at once are worth being spent this way. Not to mention that writing tests is crazy boring and it would be hard to keep a coder motivated on this project . Also, unit tests work best when they are written by the same person who writes the code tested.

IMHO the best way to gradually introduce test-driven development in PS would be to assign to somebody the project of setting up the unit testing framework with a simple example. Then it is simply a matter of changing how contributions are accepted. When someone writes a patch with a new feature to be reviewed, a dev can ask him to add few tests about it. When a bug is fixed, a test can be written to ensure that there won't be any regression with future contributions. This way also the critical existing part of PS code would gradually be tested.

As a final consideration, PS is in the process of porting to a different engine (and a different repo). I think this is a great opportunity to start pointing the project in this direction. Unreal Engine also has its own testing framework https://docs.unrealengine.com/latest/INT/Programming/Automation/TechnicalGuide/index.html

5) Eredin said it all .

I'm currently doing a big code refactoring of the music code, but once I'm done I could volunteer to help with github migration or with setting up the testing framework in the new repo in case you guys would be interested.

[Eonwind]

I'm currently doing a big code refactoring of the music code, but once I'm done I could volunteer to help with github migration or with setting up the testing framework in the new repo in case you guys would be interested.

that would be great 

[redhound]

Sounds promising!

[Jilerel]

1) In general, I agree with Eredin on this one. Git is great to implement the feature branch workflow, but given the size of the team and the fact that each developer works on a single feature, this wouldn't provide a great benefit right now. But it would make sense to switch to git if we decided to completely change platform and migrate the code to github (as suggested by redhound). Github has the advantage of making the "fork and pull request" workflow very easy which IMHO fits very well the current PS development process and adds few advantages:

• It's clean: instead of creating a lot of branches in the main repo, the prospect/anybody can create a copy of the repository in its own github account, and if he drops the task he doesn't leave dead branches anywhere.
• The pull request is basically a work-in-progress patch. Github creates an open discussion associated with each pull request and anybody is able to check the current changes in the code and suggests modifications to add to the pull requests on-the-fly. Once everything is ready, a member of the official team can merge the pull request into the main repo.
• The main repo remains clean. Each feature requires several commits, and (especially for complicated features) there can be commits that introduce bugs and break the build of the main repo. The pull request is merged only at the end, so this possibility is greatly reduced.
• The project would never miss a bit of a contribution. Working on its own online repo, a prospect can make as many commits as he wants instead of creating big monolitic patches. If he drops the task, those small contributions are not lost because github creates a local copy when a pull request is submitted, so they can be merged and picked up by someone else.

Moreover

• On github there is a fully-featured, easy-to-use issue tracker integrated with the repository.
• I don't have numbers, but I have been involved in several projects using svn and git, on sourceforge and github, and some developers tend to stay away from svn simply because they don't like it. Moreover, I have the feeling that the developers community is gradually moving to git and github in particular, so all the "kool kids" hang out there .

To sum up, I think moving to git not only would speed up and simplify the workflow, but also that it would be better suited to attract new developers (which is IMHO the killer advantage considering how much lack of "manpower" is slowing down PS ).

Ah, and as Serneum pointed out, moving an existing svn repo to git is very easy https://help.github.com/articles/importing-from-subversion/.

Agreed. In fact, everyone could contribute to the game more easily with git.

[redhound]

So, any news about changes in PS development process? Maybe there are some planned changes in moving source code or bug tracker to the new platform, using continuous integration, etc.?

[redhound]

Just want to suggest the new item to the list of services to consider. The GitLab.com SaaS looks very promising - along with unlimited number of free private or public git code repositories with issue tracker and wiki, it also offers unlimited number of team members and free CI services.

For those who don't already know, Neeno is hosting there PSExtended client and also the last compatible version of the CrystalSpace 3D engine with PlaneShift.

[Ravna]

TLDR: No updates.

As you may or may not already know, the development team has gotten a little smaller still, and half of the effort is probably going into unreal, not the current SVN code. I personally don't see any advantage of using GIT over SVN for the current team in the current situation, though it might happen with unreal (yep, speculation).

As for the unit tests, it's probably really simple, as long as no one writes them, we won't get them. Yes, there are advantages to them, but just like documentation, unit tests are hard to come by on a volunteer project.

Integrated bug trackers are really nice, but even they need someone to set them up, migrate all old stuff, etc. Currently, I don't see that happen for the current version of PS, that would also be something that could be set up a whole lot easier when we migrate to unreal (and would be starting on a whole new set of bugs/features anyway).

As it was said before, if anyone wants to contribute anything to the code, you are more than welcome to do so. If you need any additional information regarding that, come find me on IRC.

[Volki]

As you may or may not already know, the development team has gotten a little smaller still, and half of the effort is probably going into unreal, not the current SVN code.

Is this the same for future art assets? I just spent a week relearning Blender and am going to jump back into it, but I would really prefer optimizing assets for Unreal rather than something the game is going to switch away from eventually.

[Eonwind]

Is this the same for future art assets? I just spent a week relearning Blender and am going to jump back into it, but I would really prefer optimizing assets for Unreal rather than something the game is going to switch away from eventually.

Blender is a tool both CS and unreal can import from. The big real difference with unreal working with art models and animation is the pipeline to import them in game is shorter and there are more powerful tools work with them after they have been imported.
But still until there will be a working demo or version of PS on unreal everything made with unreal tools will be miles away from being usable on PS.
Therefore my suggestion is using blender and working for current PS, any art made developing with it will be usable by the new PS unreal engine anyway and it won't be trashed.

[redhound]

Let's hope the migration to Unreal will also brings some freshness into the development process also!

As you may or may not already know, the development team has gotten a little smaller still, and half of the effort is probably going into unreal, not the current SVN code. I personally don't see any advantage of using GIT over SVN for the current team in the current situation, though it might happen with unreal (yep, speculation).

Well, that depends on what we call "the team". Moving code to some kind of git-enabled public service and allowing peoples outside the core team to make pull requests will effectively make "small team" somewhat larger (an example: https://github.com/OpenMW/openmw/graphs/contributors). And it's all about the code, so there are always things to make better: https://github.com/arx/ArxLibertatis/commits/master - always, I really mean it.