[done] SMTP Authentication using OAuth2 login

[Kukulkan]

Hi. I did a commercial PureBasic program that sends a lot of emails using libcurl directly. As we started in 2012, the PB functionality regarding network was not sufficient at all and the builtin libcurl was outdated. So we use libcurl directly as a dll until today.

That was all fine since January this year, where MS announced that they want 2FA (two factor authentication) for most Office services. So the first customers comming up being unable to send emails any more. The thing is, that the SMTP protocol there only allows AUTH LOGIN XOAUTH2 as login option, which forces me to use the CURLOPT_XOAUTH2_BEARER option very soon.

Sadly, I don't find where to get the needed URLs and what I have to do to get the OAuth 2.0 Bearer Access Token that is needed. I learned that I have to let the Webbrowser open a URL (a) and then one or more token are generated (b).

Sadly, due to the fact that the same is for google and other email providers, I struggle at several points:
a) What URL to call? I mean, I can't hold a list of URLs for all email providers in the world? But I only have the users email address and password.
b) How do I get the tokens from the webbrowser URL that was opened to authenticate the user? Or is there some ID used between the URL call and some API?

I believe there is some API and there must be some reference implementation that is the same for Google, Microsoft and others, but I don't get it. I find php and java snippets for partial solutions on StackOverflow but no guide and introduction about how to implement this as a developer. Maybe some step by step explanation about what to do?

Maybe someone here already did an implementation and can help me? Or is there some good page to learn that?

[Kukulkan]

Hello. Please do not answer. In the meantime, with a lot of research, I managed to do OAuth2 authentication for our SMTP accounts with Google and Microsoft.

I would share the include, but as we're using our own libcurl include and some more functions from our function library, it does not make any sense. Sorry.

[Fred]

If you can share what you did, even small code/info can help other people to get on the right track.

[dige]

I'd be interested in that too.

[Kukulkan]

I know that the below code is using some functions that are not part of this post. Please note that I cannot post these. But in general, it is logging with a function GlobalLog(). Also, we utilize GetURLPiece(), which is similar to PB GetURLPart(), but we were not allowed to use any PB libcurl funtions and therefore replaced it by our version. You very likely can use GetURLPart() as replacement.


1) Register your app
Register your app at the corresponding email provider. For google you go to https://console.developers.google.com and for Microsoft you go to https://entra.microsoft.com. You need to register your OAuth2 using application there to get a client-id.

While registration you need to enter a redirectURL where the page indicates the result of the login. This is done by calling an URL. If your PureBasic app is a local one, this has to be a local port opened by your app. We decided for http://localhost:8100/myapp.

2) Get all needed values and urls.

We did it like this:

Code: Select all

Structure authData
  clientID.s
  clientSecret.s
  authURL.s
  tokenURL.s
  redirectURL.s
  scopes.s
EndStructure

Global NewMap authData.authData()

; Currently supported auth methods of this include:
#OAUTH_MICROSOFT = "MS"
#OAUTH_GOOGLE = "GOOGLE"

; App is configured at https://entra.microsoft.com
;
; Notes:
; As an example, Microsoft Graph API has defined permissions to do the following tasks, among others:
; * Read a user's calendar
; * Write to a user's calendar
; * Send mail as a user (!)
; I added "Microsoft Graph" permissions for User.Read and SMTP.Send
authData(#OAUTH_MICROSOFT)\clientID = "796c..."
authData()\clientSecret = "" ; Not needed for MS as we're a "public client"
authData()\authURL      = "https://login.microsoftonline.com/common/oauth2/v2.0/authorize"
authData()\tokenURL     = "https://login.microsoftonline.com/common/oauth2/v2.0/token"
authData()\scopes       = "https://outlook.office.com/SMTP.Send offline_access" ; space separated!
authData()\redirectURL  = "http://localhost:8100/myapp"

; App is configured at https://console.developers.google.com
;
; Notes:
; I added Gmail API to the list of APIs
authData(#OAUTH_GOOGLE)\clientID = "5990....apps.googleusercontent.com"
authData()\clientSecret = "GOCSPX-..."
authData()\authURL      = "https://accounts.google.com/o/oauth2/v2/auth"
authData()\tokenURL     = "https://www.googleapis.com/oauth2/v3/token"
authData()\scopes       = "https://mail.google.com" ; space separated!
authData()\redirectURL  = "http://localhost:8100/myapp"

Please note that the scopes are important. Due to the specification, you are only allowed to use the minimum of scopes that are required for your job.

Important: For SMTP login with google, some might think that the scope "https://www.googleapis.com/auth/gmail.send" might be sufficient. I learned that this is not working for SMTP. You need "https://mail.google.com/", otherwise it will not work.

3) Combine and open a URL
Combine and open a URL and run it (eg RunProgram(), use 'xdg-open' on Linux). We did it like this:

Code: Select all

; Some global cross site forgery token
Global XSF_State.s = "xsf"+Str(Random(2147483647, 1000000000))+Str(Random(2147483647, 1000000000))

Procedure.s _oauthComposeAuthUrl(provider.s, mailaddress.s = "")
  ; https://learn.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-auth-code-flow
  Protected url.s = authData(provider.s)\authURL + "?"
  url.s + "response_type=code&"
  url.s + "client_id=" + URLEncoder(authData(provider.s)\clientID) + "&"
  If mailaddress.s <> ""
    url.s + "login_hint=" + URLEncoder(mailaddress.s) + "&"
  EndIf
  url.s + "redirect_uri=" + URLEncoder(authData(provider.s)\redirectURL) + "&"
  url.s + "scope=" + URLEncoder(authData(provider.s)\scopes) + "&"
  url.s + "state=" + URLEncoder(XSF_State.s)
  GlobalLog(#RF_LOG_VERB, "Compiled oauth url: [" + url.s + "]")
  ProcedureReturn url.s
EndProcedure

4) Open a local port to receive the answer
You need to open the redirectURL port to receive the result of the request. We did it like this:

Code: Select all

Procedure.s _oauthWaitToken(provider.s, timeoutSec.i = 120)
  Protected Result.s = ""
  Protected port.s = GetURLPiece(authData(provider.s)\redirectURL, #PB_URL_Port)
  Protected host.s = GetURLPiece(authData(provider.s)\redirectURL, #PB_URL_Site)
  Protected hostI.s = host.s
  If host.s = "localhost": hostI.s = "127.0.0.1": EndIf
  
  Protected res.i = CreateNetworkServer(0, Val(port.s), #PB_Network_IPv4 | #PB_Network_TCP, hostI.s)
  If res.i = 0
    GlobalLog(#RF_LOG_CRIT, "Cannot create server on " + host.s + ":" + port.s + " for receiving oauth result")
    ProcedureReturn ""
  EndIf
  GlobalLog(#RF_LOG_VERB, "Established listening server on " + host.s + ":" + port.s)
  
  Protected *Buffer = AllocateMemory(16384)
  
  Protected Quit.i = 0
  Protected start.i = ElapsedMilliseconds()
  Repeat
    Delay(10)
    Protected ServerEvent = NetworkServerEvent(0)
    If ServerEvent
      Protected ClientID.i = EventClient()

      Select ServerEvent      
        Case #PB_NetworkEvent_Connect
          GlobalLog(#RF_LOG_VERB, "Incomming auth client connection established")
  
        Case #PB_NetworkEvent_Data
          ReceiveNetworkData(ClientID.i, *Buffer, 16384)
          Result.s = PeekS(*Buffer, -1, #PB_UTF8)
          GlobalLog(#RF_LOG_VERB, "Received from auth client: [" + Result.s + "]")
          
          Protected out.s = "HTTP/1.1 200 OK" + #LF$
          out.s + "Server: myapp" + #LF$
          out.s + "Content-Type: text/html; charset=UTF-8" + #LF$
          out.s + "Connection: close" + #LF$
          out.s + "X-Frame-Options: SAMEORIGIN" + #LF$
          out.s + "X-XSS-Protection: 1; mode=block" + #LF$
          out.s + "X-Content-Type-Options: nosniff" + #LF$
          out.s + "X-Cache-Status: BYPASS" + #LF$ + #LF$
          out.s + "<!DOCTYPE html>" + #LF$
          out.s + "<html><body>"
          out.s + "<style>"
          out.s + "html {background-color: white;}"
          out.s + ".box {background-color: #2064a8; color: white; border-radius: 40px; box-shadow: 10px 10px 20px #777; width: 80%; max-width: 800px; padding: 30px; margin: 30px auto 0 auto; text-align: center; font-size: 22px;}"
          out.s + ".check {color: white; font-size: 100px;}"
          out.s + "</style>"
          out.s + ~"<div class=\"box\">"
          out.s + ~"<div class=\"check\">&#10004;</div>"
          out.s + "<p>Thank you! You can now close this window.</p>"
          out.s + "<p>Vielen Dank! Sie können dieses Fenster jetzt schliessen.</p>"
          out.s + "<p>Merci beaucoup ! Vous pouvez maintenant fermer cette fenêtre.</p>"
          out.s + "</div>"
          out.s + "</body></html>"
  
          SendNetworkString(ClientID.i, out.s, #PB_UTF8)
          GlobalLog(#RF_LOG_VERB, "Sent success (200 OK) answer to calling auth client")
          
          Delay(500); give it some time to work and cache before closing the connection
          
          Quit = 1
  
        Case #PB_NetworkEvent_Disconnect
          GlobalLog(#RF_LOG_VERB, "Auth client connection closed")
          Quit = 1
    
      EndSelect
    EndIf
    
    If ElapsedMilliseconds() > start + (timeoutSec * 1000)
      GlobalLog(#RF_LOG_CRIT, "Connection to " + host.s + ":" + port.s + " timed out. Nothing was received in " + Str(timeoutSec) + " seconds.")
      Quit = 1
    EndIf
    
  Until Quit = 1

  CloseNetworkServer(0)
  FreeMemory(*Buffer)
  ProcedureReturn Result.s
EndProcedure

Please note that this also returns the success page. I did not wait for the result and parsind and always state success. This might be wrong, but the window must be closed anyway. Any possible error is reported later in the app.

5) Check the result of the user authentication
Now we check the result. It looks like a URL and you can use the URLDecoder() to get the values you need. In case of success it is the "code" value (the accessToken). If there is an error, check for the "error" and "error_description" values!

Also, check the returned "state" value. It must be the same as the one that was generated and submitted (see XSF_State.s valiable above)!

6) Get bearer code
With the accessToken from the call you now need to prepare a POST call to get the bearer token. We did this using libcurl directly, thus I do not have working PureBasic code for you. But maybe it helps to paste the code anyway:

Code: Select all

Procedure.s _oauthGetCode(provider.s, accessToken.s, refreshToken.s = "")
  Protected url.s = authData(provider.s)\tokenURL
  Protected newRefreshToken.s = ""
  
  Protected params.curlParams
  Protected *ctx.curlCtx
  CURL_setUrl(url.s, @params, *ctx)
  params\postFields("client_id")\value = authData(provider.s)\clientID
  params\postFields("redirect_uri")\value = authData(provider.s)\redirectURL
  If refreshToken.s <> ""
    params\postFields("refresh_token")\value = refreshToken.s
    params\postFields("grant_type")\value = "refresh_token"
  Else
    params\postFields("code")\value = accessToken.s
    params\postFields("grant_type")\value = "authorization_code"
  EndIf
  If authData(provider.s)\clientSecret <> ""
    params\postFields("client_secret")\value = authData(provider.s)\clientSecret
  EndIf
  
  GlobalLog(#RF_LOG_VERB, "Requesting bearer code using received access token...")
  Protected ret.i = CURL_DoRequest(@params)
  If ret.i <> #CURLE_OK
    authLastRefreshToken.s = ""
    ProcedureReturn "ERROR: Failed posting bearer token request (ec " + Str(ret.i) + ")"
  EndIf
  Protected json.s = Trim(params\response\body)
  If Left(json.s, 1) <> "{"
    authLastRefreshToken.s = ""
    ProcedureReturn "ERROR: Failed posting bearer token request (got no json result, please check debug logging)"
  EndIf
  
  Protected j.i = ParseJSON(#PB_Any, json.s)
  
  Protected v.i = GetJSONMember(JSONValue(j.i), "access_token")
  If v.i = 0
    authLastRefreshToken.s = ""
    ProcedureReturn "ERROR: Failed posting bearer token request (got no token. Instead, got ["+json.s+"])"
  EndIf
  Protected token.s = GetJSONString(v.i)
  
  v.i = GetJSONMember(JSONValue(j.i), "refresh_token")
  If v.i <> 0
    newRefreshToken.s = GetJSONString(v.i)
    If newRefreshToken.s <> ""
      GlobalLog(#RF_LOG_VERB, "Received refresh token: [" + newRefreshToken.s + "]")
      authLastRefreshToken.s = newRefreshToken.s
    EndIf
  EndIf
  
  GlobalLog(#RF_LOG_VERB, "Received bearer token: [" + token.s + "]")
  ProcedureReturn token.s
EndProcedure

As you can see, we use both the "access_token" and the "refresh_token". The thing is, that with the refresh_token you can get a new bearer token the next time without doing all the previous steps. Just the POST call with the refresh_token can provide you a new bearer token. It is up to you to make this a working interface

7) Send your email
As we send using libcurl directly, again there is no working PB code. But we did like this:

Code: Select all

If bearerToken.s <> ""
  ret = curl_easy_setopt(loc_curl_Handle.i, #CURLOPT_HTTPAUTH, #CURLAUTH_BEARER)
  If ret <> #CURLE_OK
    ProcedureReturn "Failed to set OAuth2 authentication in cURL (Error "+Str(ret)+"). Please check with your support."
  EndIf
  getSBStringCopy(bearerToken.s, sbBearerToken)
  ret = curl_easy_setopt(loc_curl_Handle.i, #CURLOPT_XOAUTH2_BEARER, @sbBearerToken)
  If ret <> #CURLE_OK
    ProcedureReturn "Failed to set OAuth2 bearer token in cURL (Error "+Str(ret)+"). Please check with your support."
  EndIf
EndIf

I have no idea if this is possible with the current available curl functions of PureBasic?

I hope this helped you to get your first steps...

[Fred]

That's quite some info, thanks !