So far I haven't seen a working rewriter of /foo/bar into /foo/bar.php, so I created my own. It does work in top-level directory AND subdirectories and it doesn't need hardcoding the RewriteBase.
.htaccess:
RewriteEngine on
# Rewrite /foo/bar to /foo/bar.php
RewriteRule ^([^.?]+)$ %{REQUEST_URI}.php [L]
# Return 404 if original request is /foo/bar.php
RewriteCond %{THE_REQUEST} "^[^ ]* .*?\.php[? ].*$"
RewriteRule .* - [L,R=404]
# NOTE! FOR APACHE ON WINDOWS: Add [NC] to RewriteCond like this:
# RewriteCond %{THE_REQUEST} "^[^ ]* .*?\.php[? ].*$" [NC]
