I understand that it is easy to restrict what functions and variables a particular script can access in Lua. However, for in-game scripting, even more control would be good.

Has anyone looked into limiting the amount of time a lua script can execute and the amount of memory that the script can consume?

Yes, this has come up very many times on the Lua mailing list – I'd recommend just searching there for "time limit", "instruction count limit", or things like that.

One way to solve it is to add a debug hook on number of instructions, as a proxy for execution time.