Extend/merge with google_secrets module

@dakku

• I agree a service would be a good choice - a previous implementation did just that and it's only different now because I wasn't sure of the overall way forward.
• I wondered about storing the detail in yml files as well, but it does enforce a particular implementation, and having got my head around plugins, it is I believe better with plugins and not particularly complex.
• Having a file means the user can use the $client->setAuthConfig() on that file without issue; other service account options are more complex for the client. See https://cloud.google.com/docs/authentication/getting-started
• Keeping a plugin for the storage of client secrets means there is a need to decide which implementation is being used, which for the moment I have put in yml config, but perhaps should be in settings.php for safety.
• The existing GoogleApiClient class needs to be adapted to permit both oAuth and service account access - mostly getClient() but some other places.

A medium term goal for google_calendar is to support more than one active account (currently more than one calendar, but all for the same account). This requires maintaining more than one authorization key.

A long term goal for google_calendar is to permit both service account access (as now) and oAuth access. The latter requires the oAuth capabilities of your module, additional config and UI elements, and additional logic around refresh and update, and more.

It therefore seems plausible to me that a site could want both several service accounts and several oAuth accounts usable at the same time. I'm not sure how best to do this.

So, an overall feature set:

• Service to return a Google_Client object, either unauthenticated, or authenticated by either oAuth or service account credentials
• Service to return a default authentication method (oAuth or service account), with options:
  • a method per module requesting (google_calendar / google cloud / ... )
  • by yml configuration somehow
• ??other 'glue' code needed for clients to effectively use, e.g. a field definition that can be attached to entities?
• UI / form code to select default method(s) and desired storage
• UI / form code to report status (e.g. via site status report) : include which accounts are configured, etc
• drush commands to verify account access, refresh tokens, report status, ...
• more?

Bump?
Is what I put in #3 useful, or perhaps not what you meant?

@dakku, thanks. I've just pushed further updates to google_secrets which somewhat firms up my view/needs of it.

@dakku, hope your project is going well. Thought I would update you on my progress: the google_secrets module is now alpha3 and getting reasonably stable now. Beta is in sight. I have google_calendar working well now and have issued alpha1; alpha2 is close and probably beta is in sight though tests are needed.

I would like to discuss with you how the modules could be merged - it makes no sense to have two! How do you feel about the approach I have taken?

@sadashiv, I agree we should only have one google auth project, and @dakku is I believe in general agreement too, and I am happy to merge google_secrets into google_api_client (i.e. killing off google_secrets). Equally I am happy to merge into GAuth if that is best; basically, whatever works and preferably just one of it!

However, the main reason google_secrets exists is as a wrapper to store the sensitive API access information, as well as the more obvious need for a way to create a Client from a service. I would argue that storing this sort of info in a code repo is not secure: too many "private" keys are now public by that route. I wanted to do something that would work even for multi-headed sites without shared filesystems, or by using redis or memcache, or a SQL table, or anything else. The code's main flaw is that at present it must return a filename, not the data in the file: this is a G-API legacy and would ideally be worked around.

The point of the plugins is extensibility: I have provided two which made sense to me: a static file "somewhere" visible to php, and a Drupal-managed private file. Deployment could be as simple as sticking the file in the code repo (insecure but handy), copying it around in the same way as the site's other private files, or putting it in a database with an as-yet-unbuilt plugin that extracted it on the fly.

I am not sufficiently aware of the capabilities of split config, etc, to know if that would solve the problems securely, so I rolled my own.

Moving on:

Given we are where we are, I think the best option for google_calendar users is to get to a point where google_secrets is for a while a wrapper for , so a new release can update google_secrets to do that, and also "require" whatever other module/library is needed to do the job. At some later point a new version can then eliminate the wrapper.

As for the nature of :

- A goal for google_calendar is to support multiple active authentications at a time - eg. to enable calendars from multiple service accounts.

- A long-term goal for google_calendar is to support OAuth2 security, in some far-off world that google_calendar can directly manipulate user calendars (rather than just service calendars). Dealing with OAuth2 is what made me aware of google_api_client in the first place.

- Obviously there are non-calendar uses of the API, and those should be supported. Q: should we have Drupal 'services' for each of the GAPI services? Part of me says "yes" (witness google_secrets being able to create a Google Client Calendar object) but part says "that's a lot of services".

- I am also aware that the smtp module has a sword of Damocles over it, because Google wants people to use OAuth2 to login to send mail to its mail servers. There is a sandbox version of the module that works that way but nobody has yet stepped in to agree changes and merge a patch to get it official. It would make some sense, perhaps, to try to get in there, too?

When I looked at google_api_client code before (and it was a while), there is quite a difference in structure. I think the first step in any merge would be to come up with a common structure of the result which would satisfy all users, then figure out how to get there from here. I don't think I can help without that first step.

So, I suggest that the needed thing is a "workshop" on the topic. Would @sadashiv and @dakku be up for that? Can I suggest inviting @TrevorBradley to the mix, as the guy who did the smtp sandbox? Is there anyone else who has an interest?

As an afterthought, One of the things I did in google_calendar was a wizard to guide users through the complex steps needed to make a service account. A lot of it would be the same for a non-calendar user. Should that wizard be moved into a future google_api_client module?

@sadashiv, the secrets module provides an extensible (via plugin) way to store and access the credentials information required to use a google service account. Currently the plugins in use enable that by storing the file and enabling it to be read.

I have had a look at GAuth and I can see a lot of work done, though (forgive my ignorance) it looks like it is geared to OAuth accounts. Service accounts have differences, in particular there is no linkage to any Drupal user account and there are no callbacks.

Do you think GAuth can be extended to support service accounts? I think if that is so I am amenable to switch to GAuth.

I do think a group conversation would help.... would it be best to use doodle.com to figure out a time to do that? The reason for a chat would simply be to agree the structure of the way forward.

SMTP is at drupalorg/project/smtp and the issue was:

TrevorBradley's comment

Are you talking about this in another Issue that I can see and join in with?

One thing I feel is very necessary for google_calendar is that there is a proper and preferably automated migration in place. I hope that can be arranged?

The primary link for google_secrets / google_calendar is a named service which returns an authorised and instantiated Google_Service_Calendar structure. At present this service throws exceptions if something fails, which was not my desire but is as it is.

Hi, I've had a look at 3.x now. Been very busy working on being a magazine editor and keeping the garden from invading!

I can see from the branch release notes in the how loading a configured credential, but there is little on how to configure and create the persistent credentials from which that is done.

I presume that you have dropped google_secrets idea of a plugin, and rely on config entities rather than the raw json file.

What I can't see is what I would replace the old plugin code with in google_calendar's settings or my wizard form? I created the wizard because getting a client and registering it was decidedly not simple, and it took me several hours to decode what was needed and that what I had done was the thing I was expected to do.

I can see some things in google_api_client's settings form about client credentials; is this what I am looking for, as it seems to suppose there is only one credential for the whole site?

Hi, I have been dipping in and out of trying to get ahead of the other things I'm doing.

I have had some problems with the module because it crashes on my default-for-composer Drupal 8.9 config. I am using a /web/ directory configuration, and using composer to include the Google api rather than the libraries module. The code in _google_api_client_read_scope_info() assumes I think that the 'vendor' directory is within the webroot, but for the default composer config it is not.

I wrote this to try to return the correct google client library directory in my attempt to work out the location of the google api module:

function google_api_client_load_library($return_path = FALSE) {
  if (class_exists('Google_Client')) {
    // Composer install so no problem with lib.
    if ($return_path) {
      $reflect = new ReflectionClass('Google_Client');
      $client_php = $reflect->getFileName();
      // TODO: This fails on Windows (dir sep).
      $strip = "/src/Google/Client.php";
      $dir = '';
      if (strpos($client_php, $strip) !== FALSE) {
        $dir = str_replace($strip, '', $client_php);
      }
      return $dir;
    }
    return TRUE;
  }

  $library = \Drupal::service('library.discovery')
    ->getLibraryByName('google_api_client', 'google-api-php-client');
  if ($library) {
    $php_files = array_keys($library['php']);
    $library_exists = file_exists($php_files[0]);
    if ($library_exists) {
      require_once $php_files[0];
      if ($return_path) {
        return 'libraries/google-api-php-client/';
      }
      return TRUE;
    }
  }
  return FALSE;
}

but that still leaves the first lines of _google_api_client_read_scope_info(). I have amended it as follows but it's a horrible hack:

function _google_api_client_read_scope_info() {
  $path = google_api_client_load_library(TRUE);

  // TODO: Hack: the services are in a directory parallel to the client, with "-services" appended... so this kinda works!
  $path .= '-services/src/Google/Service/';

  $files = scandir($path);

The standard version is broken because even if $path is set correctly by the google_api_client_load_library() call, it won't have a 'vendor' subdirectory -- it is itself a grandchild of the vendor directory. I would expect this code could use ReflectionClass::getFileName() for the class Google_Service to identify the location of the services module, and thus the likely location of the other services classes. This would avoid the need to assume a vendor directory layout (which IMO is necessary).

However this all feels wrong... Is there no API in the Google Services to say 'enumerate available/enabled sub APIs'? I would hope that such a function would avoid the need for any scandir() or reflecting classnames, and for the $names array as well. Then a "class_exists" call determines if it's already loaded or if a library load is needed.

Further thoughts, now I have worked through the crash.

- DX: I have found the class name GoogleApiServiceClientService somewhat confusing, Might it be possible to rename it?

- DX: I would like to be able to interact with the Google_Client class before it is used for authentication, e.g. to set the application name and/or the guzzle client options. This doesn't seem to be possible.

- DX: The Google**Interface classes should be in the same part of the namespace as the main class (i.e. ...\Entity)

- UX: In the client UI I can enter the various settings (client id, client secret, etc) for the config item, but trying to authenticate fails, with a message that suggests it's trying to do it as a personal account, not a service account.

- UX: Is there a place where I can upload the google auth json file to define the auth parameters, rather than opening the file and pasting in? I didn't find it if so.

I have now pushed a branch 'drupal9' to the google_calendar project's git which has the current state of things. Basically it's working, but the config wizard has not been modified, and neither has the command line tools. Docs work is needed too.

I have tried to include an update hook to import previous config into a new config entity, but its not tested and probably broken :( ... would appreciate feedback.

Getting to releasable state (i.e. not in a feature branch) requires:
- finishing wizard, command line, docs on my part;
- addressing the issues noted above with composer built sites (basically, make use of libraries module optional);
- ... and probably some tweaks and adjustments not yet noticed.

-I use GoogleApiClient for OAuth accounts and GoogleApiServiceClient for Service accounts which is why I kept the naming accordingly, I prefer to keep it the same so that 20+ developers trying it don't get error in the next upgrade.

I was referring specifically to "GoogleApiServiceClientService" -- I don't understand the reason for/meaning of the second word "Service".

The upgrade path if you did change it was to declare a class alias, so the existing name is an alias for whatever new name was used.

-It is possible to work with the Google_Client object eg:

But you cannot access it before the authentication has happened (as part of setGoogleApiClient), so if you want to set the application name, or configure guzzle differently from standard (as I do), you're stuck, and it is too late to do that afterwards, because if you needed it at all, you need it for the first call.

I wondered if the way to make this happen is to supply a Google_Client object as a second argument to setGoogleApiClient() that will be used instead of the one created within that function, which would be pretty simple to do as a default argument.

-Do you mean that the service class like Google_Service_Calendar should be at \Google_Service_Calendar or do you mean some other classes, can you give some example?

No. I was referring to GoogleApiClientInterface being in the same namespace as GoogleApiClient, GoogleApiServiceClientInterface being in the same namespace as GoogleApiServiceClient, that is, in the \Drupal\google_api_client\Entity namespace.

-We don't use OAuth client but use the service account only, so basically there are two ways provided by the module and we stick to Service account for google calendars and not use the OAuth Client

That was partly my fault - I didn't find the form to add a service account entity and only discovered adding the normal oauth entity. Perhaps some additional top-of-form "blue-button" links would make it clearer (as I have done with the google_calendar forms).

-I'll think about it as a improvement in future version, currently I have only the copy paste method.

That's fine - not a big deal for programmers, but much better for site builders.

For #24 I'll check and improve the library detection logic if it is done in different folder than vendor or something as you mentioned in #22

Thanks. For me, being able to use the module with the google/apiclient code being pulled in by composer into vendor is critical.

This is a sample of the directory tree created by the recommended drupal composer project

root/composer.json
root/composer.lock
root/config
root/config/sync
root/vendor
root/vendor/autoload.php
root/vendor/google
root/vendor/google/apiclient
root/vendor/google/apiclient/src
root/vendor/google/apiclient/src/Client.php
root/vendor/google/apiclient-services
root/vendor/google/apiclient-services/src
root/web/
root/web/index.php
root/web/core
root/web/modules
root/web/modules/contrib
root/web/modules/contrib/google_api_client
root/web/modules/contrib/google_api_client/src
root/web/modules/contrib/google_api_client/google_api_client.info.yml
root/web/libraries
root/web/themes
...

Sorry, multiple things on the go at once, not least one of my main servers' PSU died dramatically and took the MB and CPU with it :(

I would definitely appreciate input if you can try what's there out, but perhaps leave it a day and I will try to schedule time tomorrow to integrate the changes you made. Thanks for the reminder by the way.