cmodule 0.1.1: sneak peek at a dynamic module loading system designed for Codea

Code Sharing
41

@Andrew_Stacey

“I also like cimport being in the global namespace. I realise that you don’t like polluting the global namespace, but there’s a point at which that principle crosses the boundary and just starts making everything complicated.”

Ah, my reasoning for this wasn’t because I was concerned about polluting the global namespace in this case, it’s because I don’t like th fact that there are units of code with the same name that behave slightly differently depending on where they are used, which is confusing. Since cmodule.import is really only necessary from the Main tab, I don’t this this would actually be too cumbersome, but of you disagree with this point then I’ll consider leaving _G.cimport/cload.

“but a module can get round this by using the __pathto or __proj. If I ever use these (which at the moment I doubt I will) then I’d probably rename them to something a bit more friendly.”

I’d also rethink renaming these if you do use them. I tried to choose names that were obvious, but were unlikely to clash with variables used by the end programmer. Again, commonly, I suspect that these won’t be used often, and I think they should stand out. Additionally, if you use a modified version of cmodule, the code you write with it will not be compatible with the stock cmodule, which makes it difficult to share your code with others, which sort of defeats one of the main benefits of using a module system. I am all ears though on suggestions for more friendly names, as long as they are names that are unlikely to be used by end programmers.

Which brings me to my last point: if I were to add the search path functionality in the way I specced out above, would you use the stock version of cmodule in that case? I’d really like to avoid having alternate versions floating around, because it will make it hard for me to provide support when I can’t trust that everybody is using the same code.

EDIT: also, I still see calls in your example that look like:

cimport "Library CModule:Colour"

Does this still work? What does it do? Is it still necessary?

42

@toadkick On that last (and most important) point: Yes.

I do think that the search order ought to always be (for a module without an explicit path given):

  1. Running project
  2. Stuff in the search path
  3. Containing project

(This is what I said after you outlined the spec of a search path; I’m not clear from the ensuing comments whether or not you agreed with me.) The key is that stuff in the running project should always have priority over everything else.

With regard to the different behaviours of cimport then I don’t have much to offer. The way I’m using it then I’m not using that different behaviour so from my perspective, having the same name inside and outside makes sense and makes life (slightly) easier. But not enough that I’d refuse to use the stock code! I can always put cimport = cmodule.cimport at the start of setup and be done with it.

Similarly with regard to __pathto and __proj then I don’t anticipate using them, which is why I didn’t bother to provide better names for them in my modifications.

43

@Andrew_Stacey: Okay, excellent :slight_smile:

On the search path order:
I still didn’t quite get what you were saying before, but this spells it out clearly for me, thank you! So, I think there might still be one issue with this, given this scenario:

I use your library code. Your library has a module named “foo”, which you are importing using cimport(“foo”). My code that uses your library also (by coincidence) has a module named “foo”. Now, your library has just accidentally loaded my module.

So: I agree that there does need to be some sort of overloading mechanism, but the API needs to be carefully thought out, to prevent the above scenario.

Initially, I was not using the overridden cimport/cload closures for modules, but requiring modules to use the global cimport like so:

local myModule = cimport(__pathto "MyModule")

I’m thinking that either a) we should go back to making this how you load a module from the same project as the module owner (in which case I’d be more amenable to renaming __pathto, __proj, and __file, though I’m still not sure how to name them so that they won’t easily clash), or b) we should provide closures with a different name for loading from the same project (mimport/mload?).

a) has the downside that it requires more typing for what (I think) is the common case, and b) has the downside that it could make the API slightly more confusing.

However, either of those solutions would intrinsically allow for the search path order you’re requesting. Which of those do you think would be better?

44

@toadkick Sorry I haven’t been able to play with this project, I’m just trying to get my stuff fixed. However, I’m really looking forward to using this, great job!

45

@Briarfox: no worries, cmodule has become less stable than I original thought it was (development-wise, not program stability-wise), so unless you want to come along for the adventure I’d actually advise holding off until I get the search path feature implemented and the API is stabilized a little more (probably 1-2 weeks). Though, you are welcome to come along for the adventure! :slight_smile:

46

@toadkick I’ll counter your scenario with my scenario:

I develop a library of stuff that I use in almost all of my projects. While working on one, I realise that I need to fix some stuff/do a bit of work on one of the core modules. So I copy it into the current project since switching back and forth between projects is a pain, and also I don’t want to break the functionality of other projects while experimenting. Then when I run my experimental project I want all modules to load the experimental version by preference, even if they are in the same project as the safe version of the module.

Given that there isn’t a vast amount of code sharing going on, I think that my scenario is much more likely to occur than yours. Moreover, in your scenario (with my version of the search) I just have to rename one file. In mine (with your version of the search), I have to keep switching projects, or do a lot of renaming.

I’ve most experience with TeX and what I describe is how it looks up its “module” files. I’m pretty sure that it’s how other languages work as well.

I’d be quite happy to have a way to switch between the behaviours. I can envision having the behaviour I’d like when developing and then switching to the behaviour you’d like when distributing. (Though trying to make it easy to switch could be a nightmare and I think I wouldn’t even try until it became obvious that it was a problem needing solving.) Either of your solutions would be fine by me and I don’t see any big reason to choose between them.

47

@Andrew_Stacey: You’ve convinced me. I will probably go the mimport/mload route, to hopefully make it clear that their behavior is slightly different than the global cimport/cload (which I will leave intact).

I will implement the search feature in the order you specify for cimport/cload and all of the cmodule.* functions as well, using your modifications as a basis. mimport/mload will simply be syntactic sugar for cimport(__pathto “someModule”). I would get rid of them altogether, but I was able to memoize the cimport/cload/__pathto closures (keyed to the owning project), which turned out to be a not-insignificant optimization, which means I think I’m okay with keeping the syntactic sugar since the performance impact is trivial.

Did I leave anything out?

48

@toadkick sounds good to me!

Now all I need to do is convince you to use Briarfox’s AutoGist for uploading …

49

@Andrew_Stacey: Ha, I’m getting around to that :smiley: He and I have the same problem…we’re actively engaged on what we’re working on. I’ll check it out ASAP.

50

@Andrew_Stacey: Actually there is one more thing I want to clarify. From your earlier post:

I do think that the search order ought to always be (for a module without an explicit path given):

1. Running project
2. Stuff in the search path
3. Containing project

So, 1 and 2 are no problem. 3 is the problem: cmodule has no way of knowing what the the module’s containing project is, unless you’ve specified an absolute path, or unless the project is found in the search path.

So, it seems to me that either #3 is a non-issue, since you can tell cmodule that you want to load from the containing project by using cimport(__pathto “SomeModule”), or there does still need to be a way to specify that you would only prefer to load from the containing project if a match is not found to the running project or a project in the search path; perhaps an optional second argument to cimport, e.g. cimport(“SomeModule”, __proj)

51

@toadkick I thought that that was what the overloading did: it inserted the containing project name. The problem I had with it was that the way that it did so effectively put the containing project first in the search list (well, and didn’t allow for any other choices) by prepending it directly to the module name (if it didn’t have a : in it). What I was experimenting with was modifying this behaviour so that it didn’t simply prepend the project name but passed it as another parameter so that inside _resolveCodeaPath then it could be accessed but used in the right place - last in the search list. That was why I was trying to pass the prefix through to cimport and so on with that rather cludgy syntax involving ....

So if you want to have 3. then it is possible but you just have to be a bit more careful with the overloading. But I’m not bothered about 3. so if you’d rather drop it for now and ponder it, I’ll still be able to make use of the stock code.

52

@Andrew_Stacey:

I thought that that was what the overloading did: it inserted the containing project name. The problem I had with it was that the way that it did so effectively put the containing project first in the search list

Right, exactly. We’re on the same page here. I’m thinking that allowing a second (optional) “fallback” parameter to cimport might be a good way to solve this. Naturally, any absolute path will fail if the module is not found there. So, let’s assume we do away with the overridden cimport/cload closures, and examine this bit of code:

local SomeModule = cimport(__pathto "SomeModule")

The __pathto mechanism gives us a way to explicitly state that we unequivocally want to load from the containing project. cmodule will attempt to do so, with no search attempted, and fail if the module is not found.

However, we could modify the cimport API to allow the optional fallback project:

local SomeModule = cimport("SomeModule", __proj)

and the order will progress in the order you specified: running project first, search path projects second, fallback (which is the containing project in this case) third. If no matches are found in the search path or the fallback a “file not found” error is thrown. Naturally, the rest of the API will be modified to behave this way as well.

53

@toadkick I like that.

54

@Andrew_Stacey, anyone else who is interested: I haven’t been able to get this out of my brain, so I hacked it together last night. v0.1.0 includes the new search path feature, as well as some minor API modifications. From the changelist:

    * added search path for locating relative module paths; search path can be
      set/inspected using cmodule.path(); pass no args for a list of projects in the path,
      or pass a variable list of projects, i.e. cmodule.path("project1", "project2", "project3").
      search path may only be set once.
    * added optional fallback project param to cimport/cload and cmodule.loaded/exists/unload;
      if a module is not found in the running project or in the search path, an attempt will
      be made to load from the fallback project if specified.
    * __pathto no longer returns fully qualified path to current module when no
      module name is specified; use __pathto(__file) instead.
    * no longer generating module closures for cload; use cload(__proj "MyModule") instead.
    * renamed overridden module cimport to cinclude; cinclude only takes a module
      name, not an absolute path; it is intended for loading from the containing project
      only (it is syntactic sugar for cimport(__pathto "MyModule"))
    * memoized module cinclude/__pathto closures, which can reduce memory
      footprint considerably, and speed up loading/importing as well by not having
      to create 2 closures per module per owning project (now it's 2 closures per owning project)
    * removed redundant cmodule.import and cmodule.load. use cimport/cload instead.

link: https://gist.github.com/apendley/5411561

My next task (barring any bugfixes/other necessary modifications) will be to start using @Briarfox’s AutoGist.

55

I’ve updated to this latest version and it is working well for me so far.

56

Since I’m already using AutoGist, I’ve used it to upload your cmodule project: https://gist.github.com/loopspace/5622444.

57

@Andrew_Stacey: That’s good news. And thanks for posting an AutoGisted version! :slight_smile:

58

@Andrew_Stacey: Oh, just noticed one thing…you have the version listed as 1.0.0. Right now cmodule is at 0.1.0. I won’t consider it a final release (i.e. 1.0) until a) we don’t have to wrap modules in block comments anymore, and b) cmodule works with exported XCode projects.

59

@toadkick Yeah, sorry about that. It’s due to the fact that at the moment AutoGist does a numeric comparison of the version to see when a new one is due so it can’t cope with things like 0.1.0. I’ve put in a feature request with Briarfox on that one! In the code itself I call it GIST_VERSION to try to distinguish between the two. Maybe I should make a note in the ChangeLog to clarify.

Or maybe I should write a more advanced comparison for AutoGist …

If you’d rather I used a different system, I’m happy to do so.

60

@Andrew_Stacey It’s okay, that’s fine, I didn’t realize the limitation.

One last update btw, I’ve added a README tab to the cmodule project, containing (what I hope is) thorough documentation about cmodule’s API and features. If you spot something incorrect or confusing, or would like me to add something, please let me know. Also if you discover (or have discovered) anything you think would be helpful to add to the “Tips and tricks” section let me know.

EDIT: also fixed a bug related to cmodule.cache, bringing cmodule up to 0.1.1.